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(57) A digital color image to be Processed is input- 
ted at a color image input step, image segments are ex- 
tracted from the input color image at an image-segment 
extraction step, and the data of the extracted image seg-. 
ments is created. Next, image-segment components in 
each extracted image segment are discriminated at an 
image-segment discrimination step, and each extracted 
image segment is subjected to zoom processing, which 
conforms to the discriminated image-segment compo- 
nent, at an adaptive zoom step, thereby creating a 
zoomed image of the input image. The zoomed image 
that has been created is displayed, outputted as a hard 
copy or delivered to a transmission line at a color image 
output step. Accordingly, a color image in which image 
segments having different characteristics are mixed can 
be subjected to excellent processing. 
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Description 

BACKGROUND OF THE INVENTION 

Field of the Invention: 

This invention relates to an image processing ap- 
paratus and method. More particularly, the invention re- 
lates to an image processing method and apparatus for 
extracting image segments having different character- 
istics from an input color image and judging each of the 
characteristics of these image segments. 

Description of the Related Art; 

Apparatuses for inputting and outputting color im- 
ages have begun to find wider use in the field of office 
automation. This has been accompanied by greater op- 
portunities for handling color images obtained by using 
an input device such as a scanner to read in an original 
of a color image the greater part of which is occupied by 
a black-and-white portion or a portion wherein the 
number of colors is limited. Such a color image shall be 
referred to as a "document image". A feature of a doc- 
ument image is that, unliko other images, a document 
image includes a mixture of image segments (such as 
character image segments and photographic image 
segments) having different properties. There are cases 
in which an image having good picture quality is not ob- 
tained even if use is made of processing techniques uti- 
lized in other fields, such as a technique for zooming or 
a technique for reducing the quantity of data. Further, 
there is increasing demand for re-utilizing digital data 
obtained by structuring the color image of an original. 
More specifically, this entails inputting, into a computer, 
characters and photographs from the original in the form 
of character codes and image data, respectively, where- 
by the inputted information can be re-utilized in, say, 
desktop publishing (DTP). 

Next, a technique for subjecting an image to zoom 
processing, a technique for reducing the quantity of data 
and a technique for image segmentation (inclusive of a 
technique for structuring a document image) will be de- 
scribed. 

[Zoom processing] 

Methods of zooming an image (inclusive of convert- 
ing pixel density) include selective processing conver- 
sion (SPC), in which each pixel of an original image is 
eliminated periodically in dependence upon the zoom 
ratio (this is image reduction) or in which each pixel is 
repeated in dependence upon the zoom ratio (this is im- 
age enlargement) (see Matsumoto, Kobayashi: "Study 
on the Effect of the Resolution Conversion on Picture 
Quality" . The Journal of the Institute of Image Electron- 
ics Engineers of Japan, Vol. 12, No.. 5, pp. 354 - 362, 
1 983), and a method of obtaining the value of a pixel of 



interest by interpolation from neighboring pixels follow- 
ing a coordinate transformation (an affine transforma- 
tion, etc.). 

Further, a method proposed by the present appli- 
5 cant is available as a method of zooming a binary image. 
This method relates to an apparatus for extracting, 
smoothing and zooming outline information of a binary 
image and involves extracting outline vectors from a bi- 
nary image, creating outline vectors that have been 
io zoomed smoothly to a desired magnification (arbitrary) 
in the state of the extracted outline vector expression, 
and reproducing the binary image from the smoothing 
zoomed outline vectors, thereby obtaining a high-quality 
digital image zoomed to the desired magnification (ar- 
*5 bitrary). 

[Reduction of data quantity] 

An example of an available method of reducing the 

20 quantity of image data when the image is stored or com- 
municated is a coding method using a discrete cosine 
transform (DCT) according to the Joint Photographic 
Experts Group (JPEG). In another proposed method, a 
color image is converted to a uniform color space, after 

2S which the image is split up using color difference. The 
shape of an segment is subjected to chain coding, hue 
is quantized coarsely segment by segment using stored 
colors, saturation is subjected to a polynomial approxi- 
mation and lightness is subjected to adaptive DCT cod- 

30 ing, whereby the image data is coded. (See The Trans- 
actions of the Institute of Electronics, Information and 
Communication Engineers, B-l, Vol. J75-B-I, No. 6, pp. 
422 - 430, 1992, June, 1992.) 

Further, an example of a method of reducing the 

35 quantity of data in a binary image is MR/MMR coding or 
coding in accordance with the Joint Bi-Level Image 
Group (JBIG). 

[Image segmentation (inclusive of structuring of a 
40 document image)] 

An example of a method of segmenting a color im- 
age is a recursive threshold-value method (see -Struc- 
turing of Color Image Information by Area Partition 

45 Processing", Journal of Information Processing Society 
of Japan, Vol. 19, No. 12, pp. 1130 - 1136, Dec. 1987). 
According to this method, processing for deciding a 
threshold value for area partitioning from the value of a 
histogram in RGB, HSI, YIQ color features and then ex- 

so tracting an area is repeated recursively with regard to 
each extracted area. Another reported method is the 
limited-color expression method. This utilizes a color 
quantizing technique in which a color distribution is par- 
titioned repeatedly until attainment of a number of colors 

55 jn which a color error cannot be visually recognized by 
the human eye (see The Institute of Electronics, Infor- 
mation and Communication Engineers, 1990, Spring, 
All-Japan Convention, D146 7-168). 
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Examples of a method of segmenting image seg- 
ments applied to a document image is one in which ar- 
eas are judged pixel by pixel using the information pos- 
sessed by neighboring pixels (see Tetsuya. Ochi: "Bi- 
level Rendition Method for Documents Including Gray- 
Scale and Bilevel Image', The Transactions of the Insti- 
tute of Electronic and Communication Engineers, Vol. 
J67-B, No. 7, pp. 781 - 788, 1984), and one in which an 
image segment is segmented by utilizing the back- 
ground of a document image based upon the property 
that an image segment is partitioned by the background, 
after which the area is subjected to judgment (see Ito, 
et al: "Parallel Field Segmentation of Free-Format Doc- 
uments", 1979 Information Processing Society of Ja- 
pan, 20th All-Japan Convention, pp. 453 - 454). Some 
of these papers also propose applying the results ob- 
tained by image segmentation to the structuring of doc- 
ument images. (For example, see Yamata, et. al: "Mul- 
timedia Document Structuring Processing System", The 
Journal of the Institute of Image Electronics Engineers 
of Japan, Vol. 19, No. 5, pp. 286 - 295, 1990). 

However, the problems described below arise in the 
techniques described above. 

[Problems with zoom processing] 

The SPC zoom method is capable of forming a 
zoomed image in simple fashion but problems such as 
jaggies arise in terms of image quality. 

Further, a zoom method using coordinate transfor- 
mation is effective for natural images, such as images 
of scenery. However, when this method is applied to 
document images, the edges of character or line draw- 
ing portions such as characters and graphs become 
blurred owing to interpolation processing. 

Further, the method of zooming a binary image pro- 
posed by the present applicant is very effective as a 
method of extracting, smoothing and zooming the out- 
line information of a binary image. However, this method 
cannot be applied directly to zooming of a color image 
or grayscale image. 

[Problem with data-quantity reduction] 

The method of reducing the quantity of data by cod- 
ing using a DCT is effective for a natural image. How- 
ever, when a document image is compressed and re- 
produced, image quality deteriorates markedly at char- 
acter or line-drawing portions such as characters and 
graphs. 

The method of segmentation and coding a segment 
also is for application to natural images and lightness is 
therefore subjected to adaptive DCT coding. As a result, 
the lightness information comes to occupy a major part 
of the total quantity of information. Accordingly, this 
method is useful with regard to continuous tone portions 
of a color image. However, coding efficiency is poor with 
regard to documents, in which black-and-white portions 



or limited-color portions occupy the major part of the 
document and the importance of lightness information 
is low. 

The coding method based upon MR/MMR or JBIG 
5 is effective when reducing the amount of data in a binary 
image. However, this method cannot be applied directly 
when reducing the amount of data in a color image or 
grayscale image. 

10 [Problem with image segmentation] 

The method of image segmentation using the recur- 
sive threshold-value method or limited-color expression 
method can be applied to portion segmenting of a nat- 

is ural image. However, the method cannot be applied di- 
rectly to a document image having image segments 
whose properties differ from those of a natural image 
(because the definition of a character segment is differ- 
ent from that of a natural image). 

20 Further, the image segmentation method (inclusive 
of document structuring) for application to a character 
image is for the purpose of processing a binary image 
or a grayscale image. It is difficult to apply this method 
to a color image in which there is much overlapping of 

25 image segments. 

Thus, the proposed zoom method, data reducing 
method and image segmentation method are often suit- 
able for natural images but satisfactory results cannot 
be expected when these methods are applied to a doc- 

30 ument image. Further, the proposed methods for appli- 
cation to document images are suitable for binary imag- 
es and grayscale images but cannot be applied directly 
to color images. 

35 SUMMARY OF THE INVENTION 

A concern of the present invention is to solve the 
aforementioned problems and provide an image 
processing apparatus and method in which excellent 

40 processing is applied to a color image in which image 
segments having different characteristics are mixed. 

According to a preferred embodiment of the present 
invention, there is provided an image processing meth- 
od comprising an input step of inputting a color image, 

*s an extraction step of extracting image segments, which 
have characteristics different from a background image 
segment of the inputted color image, from the color im- 
age, and a discrimination step of discriminating the char- 
acteristics of each image segment extracted at the ex- 

50 traction step. 

The invention further discloses an image process- 
ing method for extracting each image segment from a 
color image in which image segments having different 
components are mixed, comprising a creation step of 

55 creating a reduced image from an input color image, a 
first extraction step of extracting an image segment from 
the reduced image, and a second extraction step of ex- 
tracting an image segment from the input color image 
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using data of the image segment extracted at the first 
extraction step. 

The invention further discloses an image process- 
ing apparatus comprising input means for inputting a 
color image, extraction means for extracting image seg- 
ments, which have characteristics different from a back- 
ground image segment of the input color image, from 
the color image, and discrimination means for discrimi- 
nating the characteristics of each image segment ex- 
tracted by the extraction means. 

The invention further discloses an image process- 
ing apparatus for extracting each image segment from 
a color image in which image segments having different 
components are mixed, comprising creation means for 
creating a reduced image from an input color image, first 
extraction means for extracting an image segment from 
the reduced image, and second extraction means for ex- 
tracting an image segment from the input color image 
using data of the image segment extracted by the first 
extraction means. 

Other features and advantages of the present in- 
vention will be apparent from the following description 
taken in conjunction with the accompanying drawings, 
in which like reference characters designate the same 
or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram showing a procedure for seg- 
menting image segments in the present invention; 
Fig. 2 is a diagram showing a processing procedure 
for a case in which the present invention is applied 
to color image zoom processing in an image 
processing apparatus; 

Fig. 3 is a block diagram showing an example of the 
construction of an image processing apparatus 
which executes the processing shown in Fig. 2; 
Fig. 4 is a block diagram showing hardware for im- 
plementing the arrangement of Fig. 3; 
Fig. 5 A is a diagram showing an overview of image- 
segment data; 

Fig. 5B is a diagram showing data representing 
nodes illustrated in Fig. 5A; 
Fig. 6 is a diagram showing the detailed procedure 
of an image-segment extraction step depicted in 
Fig. 1; 

Fig. 7 is a diagram showing the detailed procedure 
of a background image segmentation step depicted 
in Fig. 6; 

Fig. 8 is a flowchart illustrating a procedure for im- 
plementing the background image segmentation 
step by processing executed by a CPU shown in 
Fig. 4; 

Fig. 9 is a flowchart illustrating a procedure for im- 
plementing an intermediate image segmentation 
step by processing executed by the CPU shown in 
Fig. 4; 

Fig. 10 is a flowchart illustrating a procedure for im- 



plementing an image-segment registration step by 
processing executed by the CPU shown in Fig. 4; 
Fig. 11 is a diagram illustrating the detailed proce- 
dure of an image-segment discrimination step de- 
s picted in Fig. 1 ; 

Fig. 12 is a diagram showing examples of distribu- 
tions of RGB values in four image-segment compo- 
nents; 

Fig. 1 3 is a flowchart illustrating a procedure for im- 
10 plementing a color/monochromatic discrimination 
step by processing executed by the CPU shown in 

Fig. 4; 

Fig. 1 4 is a flowchart illustrating another example of 
a procedure for implementing a color/monochro- 

is matic discrimination step by processing executed 
by the CPU shown in Fig. 4; 
Fig. 15 is a flowchart illustrating a procedure for im- 
plementing a continuous color tone/limited-color 
discrimination step by processing executed by the 

20 CPU shown in Fig. 4; 

Fig. 1 6 is a flowchart illustrating a procedure for im- 
plementing a continuous monochromatic tone/lim- 
ited-value discrimination step by processing exe- 
cuted by the CPU shown in Fig. 4; 

25 Fig. 1 7 is a block diagram showing the construction 
of an image segmentation unit for discriminating be- 
tween a character/line-drawing image segmented 
and a pseudo-halftone image segment; 
Fig. 18 is a diagram for describing processing exe- 

30 cuted by the image segmentation unit shown in Fig. 
17; 

Fig. 1 9 is a block diagram showing the construction 
of a data holding unit depicted in Fig. 17; 
Fig. 20 is a diagram for describing processing exe- 
35 cuted by the image segmentation unit shown in Fig. 
17; 

Fig. 21 is a diagram for describing processing exe- 
cuted by the image segmentation unit shown in Fig. 
17; 

*o Fig. 22 is a diagram for describing processing exe- 
cuted by the image segmentation unit shown in Fig. 
17; 

Fig. 23 is a diagram illustrating the detailed proce- 
dure of an adaptive zooming step shown in Fig. 1; 

^5 Fig. 24 is a diagram illustrating a sequence of image 
segments which undergo zooming; 
Fig. 25 is a diagram illustrating processing in a case 
where the present invention is applied to processing 
for compressing a color image in an image process- 

50 jng apparatus; 

Fig. 26 is a block diagram showing the construction 
of an image processing apparatus which executes 
the processing shown in Fig. 25; 
Fig. 27 is a block diagram showing hardware for im- 

55 plementing the arrangement of Fig. 26; 

Fig. 28 is a block diagram showing another con- 
struction of an image processing apparatus which 
executes the processing shown in Fig. 25; 
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Fig. 29 is a diagram showing the structure of com- 
pressed data; 

Figs. 30A and 30B are diagrams showing the form 
of image-segment data contained in compressed 
image-segment data; 

Fig. 31 is a diagram showing the detailed procedure 
of an adaptive compression step depicted in Fig. 25; 
Fig. 32 is a diagram showing the detailed procedure 
of a color image reproduction step depicted in Fig. 
25; 

Fig. 33 is a flowchart illustrating the detailed proce- 
dure of a continuous color tone reproduction step 
depicted in Fig. 32; 

Fig. 34 is a flowchart illustrating the detailed proce- 
dure of a limited-color, character/line-drawing im- 
age segment reproduction step depicted in Fig. 32; 
Fig. 35 is a diagram showing a processing proce- 
dure for a case in which the present invention is ap- 
plied to document image editing processing in an 
image processing apparatus; 
Fig. 36 is a diagram showing the detailed procedure 
of a background image segmentation step accord- 
ing to a fourth embodiment according to the present 
invention; 

Figs. 37A to 37C are diagrams for describing an 
overview of processing for judging a background 
image segment; 

Fig. 38 is a diagram illustrating the detailed proce- 
dure of a background image-segment discrimina- 
tion step shown in Fig. 36; 

Fig. 39 is a flowchart illustrating a procedure for im- 
plementing a background image segmentation step 
shown in Fig. 38; 

Fig. 40 is a flowchart illustrating a procedure for im- 
plementing a background image segmentation step 
shown in Fig. 38; 

Fig. 41 is a flowchart illustrating a procedure for im- 
plementing a background image segmentation step 
shown in Fig. 38; 

Fig. 42 is a diagram for describing a method of 

measuring area outside a background; 

Fig. 43 is a diagram showing the composition of a 

color image processing method according to a fifth 

embodiment of the present invention; 

Fig. 44 is a diagram showing the construction of a 

color image processing apparatus which executes 

the illustrated processing; 

Fig. 45 is a diagram for describing a step of forming 
a reduced image; 

Fig. 46 is a diagram showing the constitution of an 
input-image image-segment extraction step depict- 
ed in Fig. 43; 

Fig. 47 is a diagram showing the form of processing 
of an input-image image-segment extraction step 
depicted in Fig. 43; 

Fig. 48 is a diagram showing an example in which 
image-segment data extracted by the input-image 
image-segment extraction step of Fig. 43 is applied 



to the color image zoom method described in con- 
nection with the first embodiment; 
Fig. 49 is a diagram showing an example in which 
image-segment data extracted by the input-image 

5 image-segment extraction step of Fig. 43 is applied 
to the color image compression method described 
in connection with the second embodiment; 
Fig. 50 is a diagram showing an example in which 
image-segment data extracted by the input-image 

io image-segment extraction step of Fig. 43 is applied 
to the structuring of a document image described in 
connection with the third embodiment; 
Fig. 51 is a diagram showing the constitution of an 
input-image image-segment extraction step ac- 

is cording to a sixth embodiment of the present inven- 
tion; 

Fig. 52 is a diagram showing the form of processing 
of the input-image image-segment extraction step 
depicted in Fig. 51 ; 
20 Fig. 53 is a diagram showing the constitution of an 
input-image image-segment extraction step ac- 
cording to a seventh embodiment of the present in- 
vention; 

Fig. 54 is a diagram showing the form of processing 
25 of the input-image image-segment extraction step 
depicted in Fig. 53; 

Fig. 55 is a diagram for describing a group of image 
segments of the same level; and 
Figs. 56A and 56B are diagrams showing an exam- 
30 pie of the memory map of a storage medium storing 
program code means according to the present in- 
vention. 

DESCRIPTION OF THE PREFERRED 
35 EMBODIMENTS 

According to the gist of the present invention, a 
color document image in which image segments having 
different components (segmentation components) are 

40 mixed undergoes image-segment extraction and dis- 
crimination of components. An "image-segment 0 is a 
linked portion having the same characteristic in an im- 
age. By focusing on an image segment which becomes 
the background, image segments contained in an image 

45 are extracted and the components of each image seg- 
ment (hereinafter referred to as "image-segmentation 
components") are discriminated. 

An image-segmentation component indicates the 
properties of an image segment, such as whether an 

so image segment is one comprising a character or a line 
drawing, one comprising a photograph or one which is 
a color image segment. Examples of image-segment 
components are described below. Specifically, an image 
segment of a document image is described as having 

55 any of the following image-segment components, for ex- 
ample: a "continuous color tone", a "limited-color char- 
acter or line drawing", a "limited-color pseudo-halftone", 
a "continuous monochromatic tone" a "limited-value 
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character or line drawing" and a "limited-value pseudo- 
halftone". Though the details of these image-segment 
components will be set forth below, it is of course pos- 
sible to use image-segment components based upon a 
simpler or more detailed classification. 

Continuous color tone: an image segment in which 
the original is a color photograph 
Limited-color character or line drawing: an image 
area in which the original is a character or line draw- 
ing in a color other than black 
Limited-color pseudo-halftone: an image segment 
in which the original is a colorized photograph that 
has been subjected to pseudo-halftone processing 
Continuous monochromatic tone:- an image seg- 
ment in which the original is a black-and-white gray- 
scale photograph 

Limited-value character or line drawing: an image 
area in which the original is a black-and-white char- 
acter or line drawing 

Limited-value pseudo-halftone: an image segment 
in which the original is a black-and-white photo- 
graph that has been subjected to pseudo-halftone 
processing 

The segmentation of image segments in a color 
document image at the center of the present invention 
will now be described in simple terms. 

An image segment having an image-segment com- 
ponent in a document image is considered to be one 
delimited by the background. In other words, image seg- 
ments can be extracted by partitioning an input image 
at portions delimited by the background. However, while 
a black-and-white original generally has a "white" back- 
ground color, the background color of a color original is 
not necessarily white. In the case of a color original, 
moreover, a certain background often is overlapped by 
backgrounds of other colors. Accordingly, in the present 
invention, extraction of background and the partitioning 
of image segments based upon the background are re- 
peated recursively with respect to image segments (in- 
itially the input image), whereby segmentation of images 
areas is achieved. 

Fig. 1 is a diagram showing a procedure for seg- 
menting image segments according to the present in- 
vention. 

First, a background 5002 is extracted from an input 
image 5001, then image segments 5003 and 5004, 
which can be segmented using the background 5002, 
are extracted. Since no further extraction of background 
can be performed with regard to the image segment 
5003, segmentation of image segments is concluded for 
this image segment. With regard to area 5004, however, 
a background 5005 can be extracted and image seg- 
mentation is continued for this image segment. Image- 
segment components are discriminated with regard to 
each image segment thus segmented from the input im- 
age. 



An image segment indicative of background shall 
be referred to as a "background image segment", and 
an image segment indicative of something other than 
background shall be referred to as a "general image 
s segment". Further, an image segment currently being 
processed, namely an image segment containing an im- 
age segment for which judgment as to whether or not it 
includes background is incomplete and an image seg- 
ment containing a plurality of image segments shall be 
io referred to as an "intermediate image segment", and an 
image segment (inclusive of a background image seg- 
ment) for which partitioning processing has been com- 
pleted shall be referred to as a "final image segment". 
It should be noted that an input image is also an inter- 
is mediate image segment. 

An image processing apparatus according to em- 
bodiments of the present invention will now be de- 
scribed in detail with reference to the drawings. 

The image processing apparatus and method de- 
20 scribed below can be utilized in pixel-density conversion 
or zoomed output in a device, such as a color printer, 
which handles a color document image, in pixel-density 
conversion at the time of enlargement/reduction and 
output in a DTP system, and in pixel -density conversion 
25 and zoomed output in a color facsimile. Further, the im- 
age processing apparatus and method can be utilized 
in reducing quantity of data at the time of image com- 
munication in a color facsimile or in the network equip- 
ment of a LAN or the like, and when storing a color image 
30 in a storage device such as a magnetic disk device. 

<First Embodiment 

[Processing procedure] 

35 

Fig. 2 is a diagram showing a processing procedure 
for a case in which the present invention is applied to 
color image zoom processing in an image processing 
apparatus. 

40 As shown in Fig. 2, a digital color image to be proc- 
essed is inputted at a color image input step 1 and image 
segments are extracted from the input color image at an 
image-segment extraction step 2, thereby creating data 
representing the extracted image segments (this data 

45 shall be referred to as "image-segment data"). Next, 
each extracted image segment is discriminated for im- 
age-segment components at an image-segment dis- 
crimination step 3 and each extracted image segment 
is subjected to zoom processing conforming to the dis- 

so criminated image-segment component at an adaptive 
zooming step 4, thereby creating a zoomed image of the 
input image. Then, at a color image output step 5, the 
created zoomed image is displayed, outputted as a hard 
copy or outputted to a communication line. 

55 

[Construction of apparatus] 

Fig. 3 is a block diagram showing the construction 
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of an image processing apparatus which executes the 
processing shown in Fig. 2, and Fig. 4 is a block diagram 
showing hardware for implementing the arrangement of 
Fig. 3. 

The hardware shown in Fig. 4 will be described first. 
The hardware includes a CPU 2001 , a bus 2007 for in- 
terconnecting the CPU 2001 and other components 
mentioned below, a ROM 2004 in which programs exe- 
cuted by the CPU and data are stored in advance, a 
RAM 2006 utilized as the working memory of the CPU 
2001 , a storage device 2002 such as a hard disk or mag- 
neto-optical disk for storing programs and image data, 
a disk input/output unit 2003 for data input/output with 
respect to the storage device 2002, a color image input 
unit 2008, a color image output unit 2009, and an input/ 
output port 2005 for input/output of data with respect to 
the color image input/output units 2008, 2009. 

In accordance with a program stored beforehand in 
the ROM 2004 or storage device 2002, the CPU 2001 
controls each of the aforementioned components via the 
bus 2007 and provides the color image output unit 2009 
with results obtained by applying processing to color im- 
age data inputted from the color image input unit 2008 
or color image data stored in the storage device 2002, 
wherein the RAM 2006 is used as a working area. 

As shown in Fig. 3, a color image input unit 1001 
inputs digital color image data to be processed and 
stores this data in an input image memory 1006. Under 
the control of the CPU 2001 which operates in accord- 
ance with the program stored in the ROM 2004 or stor- 
age device 2002, the color image input unit 1001 inputs 
the color image data using the color image output unit 
2009 and stores the color image data in the RAM 2006 
or storage device 2002. It should be noted that the color 
image input unit 1001 may be an input unit for reading 
in an image by a color image scanner, an input unit 
which receives color image data from a communication 
line, an input unit for reading in color image data that 
has been stored in an image storage device or a com- 
bination of these input units. 

An image-segment extraction unit 1002 extracts im- 
age segments from the image that has been stored in 
an input image memory 1 006 and stores the extracted 
image-segment data in an image-segment data memo- 
ry 1007. An image-segment discrimination unit 1003 
discriminates image-segment components in each im- 
age segment extracted by the image-segment extrac- 
tion unit 1002 and stores the image-segment compo- 
nents in the image-segment data memory 1007. An 
adaptive zoom unit 1004 subjects each image segment 
extracted by the image-segment extraction unit 1002 to 
zoom processing conforming to the image-segment 
component discriminated by the image-segment dis- 
crimination unit 1003, thereby creating an image ob- 
tained by zooming the input image, and stores the image 
data representing the zoomed image in a zoomed-im- 
age memory 1008. It should be noted that information 
indicating the attribute of each image segment is stored 



in the image-segment data memory 1007. 

The image-segment extraction unit 1002. image- 
segment discrimination unit 1003 and adaptive zoom 
unit 1004 can be constructed by the CPU 2001, which 

5 operates in accordance with the program stored in the 
ROM 2004 or storage device 2002, and the RAM 2006 
used as the working area or the storage device 2002. It 
goes without saying that the units 1002, 1003 and 1004 
may also be constructed of there own CPUs, RAMs and 

io storage devices and may be implemented by their own 
hardware. 

A color image output unit 1 005 outputs the zoomed 
image data created by the adaptive zoom unit 1 004 and 
stored in the zoomed-image memory 1 008. The desti- 

is nation of this output is a display, a printer or copier for 
making hard copies, a communication line or an image 
storage device. An arrangement can be adopted in 
which, under the control of the CPU 2001 which oper- 
ates in accordance with the program stored in the ROM 

20 2004 or storage device 2002, the color image output un it 
1005 outputs the zoomed image data, which has been 
stored in the RAM 2006 or storage device 2002, to the 
color image output unit 2009. 

The input image memory 1 006, the image-segment 

25 data memory 1007 and the zoomed-image memory 
1008 can also be realized by the RAM 2006 or storage 
device 2002. It goes without saying that these memories 
can be realized by their own storage devices. 

30 [Details of processing] 

The main steps shown in Fig. 2 will now be de- 
scribed in detail while using the arrangement of Fig. 3. 

35 Image-segment extraction step 

At the image-segment extraction step 2, image seg- 
ments are extracted from the color image that has been 
stored in the input image memory 1006 at the color im- 
40 age input step 1 , data representing the extracted image 
segment is created and the data is stored in the image- 
segment data memory 1007. 

The image-segment image data will now be de- 
scribed. 

« Fig. 5A is a diagram showing an overview of the im- 
age-segment data. The image-segment data expresses 
a group of image segments in an input image by a tree 
structure in which each image segment is adopted as a 
node. The parent-child relationship of the tree structure 

so represents the image-segment inclusion relationship. A 
node representing an intermediate image segment has 
child nodes which represent one background image 
segment and several intermediate image segments or 
general image segments contained in an intermediate 

55 image segment. 

Fig. 5B is a diagram illustrating items of data repre- 
senting each node, namely each individual image seg- 
ment. Each item of image-segment data is composed 
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of 'image-segment type", "image-segment position", 
"image-segment shape", "image-segment component", 
"image-segment color", "parent node" and "child node". 
The details are as follows: 

5 

Image-segment type: indicates the type of image 
segment (background image segment, general im- 
age segment, intermediate image segment) 
Image-segment position: indicates the coordinates 
(upper-left xy coordinates, width, height) of a cir- 10 
cumscribed rectangle enclosing the image segment 
Image-segment shape: the shape of the image seg- 
ment expressed by a binary image 
Image-segment component: indicates which im- 
age-segment component the image segment has is 
been discriminated as 

Image-segment color: stores the average RGB val- 
ue in case of a limited-color or limited-value image- 
segment component 

Parent node: a pointer indicating a parent node 20 
Child nodes: a plurality of pointers indicating child 
nodes 

It should be noted that a black pixel (pixel value = 
"I") of a binary image representing the shape of an im- 25 
age segment indicates an image segment, and that the 
size of the binary image and positions on the input image 
correspond to the position of a rectangle indicated at the 
position of image segment. 

Fig. 6 is a diagram showing the detailed procedure 30 
of the image-segment extraction step 2. According to 
the image-segment extraction step 2. as described 
above, an image segmentation step 20. through which 
an intermediate image segment (inclusive of the input 
image) contained in an image is further segmented into 35 
a background image segment and an intermediate im- 
age segment, is repeated recursively until completion of 
the image segmentation step 20 has been determined 
by an unsegmented image-segment search step 21. 
The result is extraction of image segments. 40 

The unsegmented image-segment search step 21 
searches for image-segment data stored in the image- 
segment data memory 1007 and searches for interme- 
diate image segments for which segmentation is incom- 
plete, namely intermediate image segments not having 45 
any child nodes. In a case where the image-segment 
data has an intermediate image segment for which seg- 
mentation is incomplete, the image segmentation step 
20 is performed using this intermediate image segment 
as an input intermediate image segment. In a case so 
where the image-segment data does not have an inter- 
mediate image segment for which segmentation is in- 
complete, the image-segment extraction step 2 is termi- 
nated. 

The image segmentation step 20 inputs an interme- ss 
diate image segment (inclusive of the input image) and 
judges whether the inputted intermediate image seg- 
ment can be further segmented into a background im- 
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age segment and several intermediate image seg- 
ments. If it is judged that further segmentation is impos- 
sible, the inputted intermediate image segment is deter- 
mined to be a final image segment and this information 
is registered in the image-segment data of this image 
segment stored in the image-segment data memory 
1 007. If it is judged that further segmentation is possible, 
then the inputted intermediate image segment is seg- 
mented into a background image segment and several 
intermediate image segments and this information is 
registered in the image-segment data of this image seg- 
ment stored in the image-segment data memory 1007. 

More specifically, if. by way of example, the image 
segment 5003 shown in Fig. 1 is an input intermediate 
image segment and it is judged that an intermediate im- 
age segment cannot be segmented from the image seg- 
ment 5003, the image segment 5003 is determined to 
be a final image segment. Further, if the image segment 
5004 shown in Fig. 1 is an input intermediate image seg- 
ment and it is judged that an intermediate image seg- 
ment can be segmented from the image segment 5004, 
the background image segment 5005 and the interme- 
diate image segment 5006 are segmented from the im- 
age segment 5004. 

A background image segmentation step 22 deter- 
mines whether the inputted intermediate image seg- 
ment has a background image segment. If the back- 
ground image segment is judged to be present, the 
background image segment is segmented. According to 
an intermediate image segmentation step 23, an image 
segment other than the image segment segmented as 
the background image segment at the background im- 
age segmentation step 22 is segmented into several in- 
termediate image segments using a geometrical rela- 
tionship. More specifically, connected pixels or connect- 
ed pixels which are nearly geometrical are adopted as 
being one intermediate image segment. According to an 
image-segment registration step 24, image segments 
segmented at the background image segmentation step 
22 and intermediate image segmentation step 23 are 
registered in the image-segment data memory 1007. 
The details of the background image segmentation step 
22, intermediate image segmentation step 23 and im- 
age-segment registration step 24 will be described be- 
low. 

Fig. 7 is a diagram showing the detailed procedure 
of the background image segmentation step 22. A back- 
ground color extraction step 211 analyzes the colors of 
the inputted intermediate image segment and judges 
whether the inputted intermediate image segment con- 
tains a color indicative of a background image segment. 
More specifically, the judgment is performed on the ba- 
sis of whether pixels having a color in a specific range 
occupy the major part of the inputted intermediate image 
segment. A binarization step 21 2 creates a binary image 
indicative of the background image segment using the 
color indicating the background image segment extract- 
ed at the background color extraction step 211. A back- 
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ground imago-segment shaping step 213 removes 
noise and small image segments from the binary image 
representing the background image segment created at 
the binarization step 212, shapes the background image 
segment and verifies, by the processing for removing 
small areas, whether the background image segment 
will vanish. 

Fig. 8 is a flowchart illustrating a procedure for im- 
plementing the background image segmentation step 
22 by the processing executed by the CPU 2001 . 

Steps S101 from S103 correspond to the back- 
ground color extraction step 211. 

More specifically, a color histogram is created in the 
working memory (RAM 2006 or storage device 2002) at 
step S101. For example, in a case where the input im- 
age is expressed in an RGB colorimetric system, the 
color histogram is three-dimensional data of RGB rep- 
resenting the frequency of occurrence of the color of 
each pixel In the intermediate image segment, namely 
the ratio of the number of pixels having a certain RGB 
value to the number of pixels of the intermediate image 
segment. It should be noted that quantization may be 
performed again when the color histogram is created. 
For example, in a case where an image of eight bits per 
R, G, B has been inputted, creating a color histogram 
while re-quantizing to four bits or seven bits results in a 
more complicated process. However, the memory size 
for storing the histogram can be reduced. An additional 
effect is that even if the color of the background image 
segment is uneven, a background image segment can 
be segmented in stable fashion. 

Further, the colorimetric system of the image to be 
processed need not be an RGB colorimetric system. For 
example, a color histogram ix created by transforming 
the input image to another colorimetric system in ad- 
vance, e.g., an L*a*b* colorimetric system, an L*u*v* 
colorimetric system or an HSI colorimetric system. In 
case the dimensions of the color histogram also use the 
dimensions of the three components of the converted 
colorimetric system. For example, in case of the L*a*b* 
colorimetric system, the three-dimensional data of 
L*a*b* is used. Though the process becomes more 
complicated by using another colorimetric system such 
as the L*a*b* system, an advantage is that a back- 
ground image segment close to human vision can be 
segmented. A case in which a color histogram is created 
in the RGB colorimetric system will be described below. 

Next, the color having the high frequency of occur- 
rence (namely a color which is a candidate for the back- 
ground color) is obtained at step 102. This is followed 
by steps 103, at which it is determined whether the max- 
imum frequency obtained is greater than a threshold val- 
ue th. If the maximum frequency is less than the thresh- 
old value th, then it is decided that the inputted interme- 
diate image segment does not contain a background im- 
age segment and the background image segmentation 
step 22 is terminated. If the maximum frequency is 
greater than the threshold value th, then it is decided 



that the inputted intermediate image segment contains 
a background image segment and the program pro- 
ceeds to step S104. The threshold value th, which is de- 
cided upon taking the properties of the input image into 
s account, is 0.5 (50%). by way of example. 

Step S104, which corresponds to the binarization 
step 212, creates, in the working memory, a binary im- 
age of the intermediate image segment binarized to the 
pixel having the color of maximum frequency and the 
io pixels other than this pixel. For example, the binary im- 
age created is one in which the pixel having the color of 
maximum frequency is made "1" and the other pixels 
are made "0"s. 

Steps S105 through S109 correspond to the back- 
's ground image-segment shaping step 21 3. 

Specifically, noise in the created binary image is re- 
moved at step S105 by applying well-known reduction/ 
enlargement processing, and small areas are removed 
from the noise-free binary image at steps S106 through 
20 S108. First, with regard to the noise-free binary image, 
a label image is created in the working memory and con- 
nected components are extracted by well-known labe- 
ling processing at step S106. The label image is one in 
which a unique number is assigned to each connected 
25 component and the numbers are made the pixel values 
of the connected components. Next, the number of pix- 
els (the area) of each connected component is obtained 
and stored in the working memory at step Si 07. Next, 
at step S 1 08, connected components for which the num- 
30 bers of pixels is less than the threshold value (namely 
connected components of small area) are eliminated. In 
other words, the values of pixels corresponding to these 
connected components of the binary image are made 
■0 D s. 

35 Next, it is determined at step S109 whether there 
are any remaining connected components (background 
image segments). If the answer is "YES", then it is de- 
termined that there is a background image segment; 
otherwise, it is determined that there are no background 

40 image segments. In a case where there is a background 
image segment, a binary image indicative of a back- 
ground image segment remains in the working memory. 

Fig. 9 is a flowchart illustrating a procedure for im- 
plementing the intermediate image segmentation step 

4S 23 by processing executed by the CPU 2001 . 

Step S201 creates an image segment other than a 
background image segment obtained by reversing the 
blacks and whites of the binary image segment indicat- 
ing the background image segment created at the back- 

50 ground image segmentation step 22, namely by chang- 
ing "1 n pixels to "0" pixels and changing "0 B pixels to "1 " 
pixels. In other words, a binary image indicating an in- 
termediate image segment in the inputted intermediate 
image segment is created. This binary image is stored 

55 jn a working memory. 

Next, at step S202, the binary image that has un- 
dergone the black-white reversal is segmented into in- 
termediate image segments and neighboring compo- 
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nents are extracted. "Neighboring components" are a 
set of closely adjacent (connection is not required) black 
pixels adopted as one component in a binary image. 
These neighboring components are adopted as an in- 
termediate image segment. In a document image, for 
example, neighboring components" refers to a column 
comprising a collection of characters that are not con- 
nected but closely adjacent. A method of extracting 
neighboring components is that reported by Ito et al. 
(see "Parallel Field Segmentation of Free-Format Doc- 
uments', 1 979 Information Processing Society of Ja- 
pan, 20th All-Japan Convention, 2E-1). 

Next, a circumscribed rectangle indicating the posi- 
tion of extracted neighboring components is calculated 
and stored in the working memory at step S203. 

By virtue of the foregoing procedure, an image seg- 
ment other than a background image segment can be 
segmented Into one or a plurality of intermediate image 
segments from the inputted intermediate image seg- 
ment and the position (circumscribed rectangle) thereof 
can be stored in the working memory. 

Fig. 10 is a flowchart illustrating a procedure for im- 
plementing the image-segment registration step 24 by 
processing executed by the CPU 2001 . In a case where 
it has been determined at the background image seg- 
mentation step 22 that a background image segment is 
present in the inputted intermediate image segment, the 
segmented background image segment and the inter- 
mediate image segment are registered in the form illus- 
trated in Figs. 5A and 5B. If it has been determined that 
a background image segment is not present, the input- 
ted intermediate image segment is registered as a gen- 
eral image segment. More specifically, it is determined 
at step S200 whether a background image segment is 
present.- If the background image segment is not 
present, the program proceeds to step S219, at which 
the image-segment type of the node indicating the in- 
putted intermediate image segment is changed to "gen- 
eral image segment' and the processing is terminated. 

If a background image segment is found at step 
S200, a new node indicating an image segment is cre- 
ated at step S211 , the image-segment type is registered 
for the new mode at step S212 and the type of image 
segment to be processed is discriminated at step S21 3. 
If the type of image segment is a background image seg- 
ment, a binary image indicating the background image 
segment is registered for the shape of the image seg- 
ment of the new node at step S214. If the type of image 
segment discriminated at step S213 is an intermediate 
image segment, position information obtained from the 
circumscribed rectangle is registered for the image-seg- 
ment position of the new node at step S215. Next, step 
S216 registers a black-white reversed binary image, 
which indicates the intermediate image segment, for the 
shape of the image segment of the new node. It should 
be noted that the image-segment position of the back- 
ground image segment is the same as the image-seg- 
ment position of the parent node (the input intermediate 



image segment). Hence, the registration of this position 
is eliminated. 

Next, the inputted intermediate image segment is 
registered as the parent node of the new node at step 
s S21 7 and the new node is registered as a child node of 
the node indicating this input intermediate image seg- 
ment at step S218. Processing is then terminated. 

Image-segment discrimination step 

10 

The image-segment discrimination step 3 discrimi- 
nates the type of each image segment extracted at the 
image-segment extraction step 2 and stores the result 
of discrimination in the image-segment data. An exam- 

15 pie will now be described in which it is determined 
whether each image segment is a "continuous color 
tone", a "limited-color character or line drawing", a "lim- 
ited-color pseudo-halftone", a "continuous monochro- 
matic tone", a "limited-value character or line drawing" 

20 or a "limited-value pseudo-halftone". 

Fig. 11 is a diagram illustrating the detailed proce- 
dure of the image-segment discrimination step 3. Ac- 
cording to the image-segment discrimination step 3, it 
is determined by a color/monochromatic discrimination 

2S step 32 whether an image segment of interest is color 
or monochromatic (grayscale or black and white), then 
it is determined whether a color image segment is a con- 
tinuous color tone or a limited color at a continuous color 
tone/limited-color discrimination step 33, and it is deter- 

30 mined whether a monochromatic image segment is a 
continuous color tone or a limited value by a continuous 
monochromatic tone/limited-value discrimination step 
34. With regard to an image segment discriminated as 
being a limited color or limited value, it is determined 

35 whether the image segment is the image segment of a 
character or line drawing or the image segment of a 
pseudo-halftone by a character or line-drawing/pseudo- 
hatftone discrimination step 35. Under the control of an 
image-segment discrimination control step 31 , the fore- 

40 going steps are repeated until there are no longer any 
undiscriminated image segments. As a result, image- 
segment components of each image segment are dis- 
criminated. 

Fig. 1 2 is a diagram showing examples of distribu- 
45 tions of RGB values in four image-segment compo- 
nents. Fig. 12 illustrates examples of RGB value distri- 
butions of the following image segments shown clock- 
wise from the upper left: a "continuous color tone" image 
segment, a "continuous monochromatic tone" image 
50 segment, a "limited-color" image segment, and a "limit- 
ed-value" image segment. 

The color/monochromatic discrimination step 32 
discriminates whether an image segment of interest is 
color (an image segment of a continuous color tone or 
ss a limited-color image segment) or monochromatic (an 
image segment of a continuous monochromatic tone or 
a limited-value image segment). More specifically, as 
shown in Fig. 1 2, by utilizing the fact that the saturation 
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of each pixel in case of a monochromatic image seg- 
ment takes on a value near 7ero (R = G = B in an RGB 
colorimetric system), a monochromatic image is judged 
to be an image segment wherein there is a single-satu- 
ration, at which the frequency in the saturation histo- 5 
gram of the image segment indicates a peak, in the vi- 
cinity of zero and, moreover, wherein there is little vari- 
ance in the histogram. 

Fig. 1 3 is a flowchart illustrating a procedure for im- 
plementing the color/monochromatic discrimination 
step 32 by processing executed by the CPU 2001 . 

At step S301 , saturation is calculated from the pixel 
values (e.g., RGB values) in the image segment, a sat- 
uration histogram is created and the histogram is stored 
in the working memory. In a case where this is ex- 
pressed in a colorimetric system which already includes 
a saturation component (S in case of HSI), as in an HSl 
colorimetric system, for example, the saturation histo- 
gram can be created directly using the saturation com- 
ponent. Next, the saturation at which the frequency of 
the saturation histogram peaks is detected at step S302, 
the variance of the saturation histogram is calculated at 
step S302 and image segment discrimination is per- 
formed at step S304 using the saturation and variance 
obtained. This discrimination involves judging that the 
image segment is a monochromatic image segment if 
there is a single saturation, which is located in the vicin- 
ity of zero (i.e., which is below a predetermined thresh- 
old value), at which the frequency peaks and, moreover, 
the variance is less than a predetermined threshold val- 
ue. The image segment is judged to be a color image 
segment in all other cases. 

Fig. 1 4 is a flowchart illustrating another example of 
a procedure for implementing the color/monochromatic 
discrimination step by processing executed by the CPU 
2001. Though the precision of discrimination is inferior 
in comparison with the procedure of Fig. 1 3, discrimina- 
tion can be performed in simpler fashion. 

A variable E is reset to zero at step S401, the ab- 
solute value (IR-GI) of the difference between the R and 
G values of a pixel of interest in the image segment of 
interest is added to the variable E at step S402, and the 
absolute value (IR-BI) of the difference between the R 
and B values of a pixel of interest in the image segment 
of interest is added to the variable E at step S403. The 
processing of steps S402 and S403 is repeatedly exe- 
cuted for all pixels of the image segment. This is judged 
at step S404. If the processing of all of these pixels is 
finished, a value (E/N) which is the result of dividing the 
variable E by the number N of pixels in the image seg- 
ment of interest is compared with a predetermined 
threshold value th at step S405. The image segment is 
judged to be a color image segment if E/N > th holds 
and to be a monochromatic image segment in ail other 
cases. In other words, according to this method, wheth- 
er the image segment is judged to be a color image seg- 
ment or a monochromatic image segment depends up- 
on the average value of the sum of the absolute values 



of R-G and R-B of all pixels of the image segment. 

Next, as shown in Fig. 1 2. the continuous color tone/ 
limited-color discrimination step 33 makes use of the 
fact that, in a case where the image segment is a limited- 
color image segment, the pixel values of the area con- 
centrate in specific values. Based upon this fact, an im- 
age segment wherein there is a single color at which the 
frequency in the color histogram of the image segment 
indicates a peak and, moreover, there is little variance 
in the color histogram, is judged to be a limited-color 
image segment. 

Fig. 1 5 is a flowchart illustrating a procedure for im- 
plementing the continuous color tone/limited-color dis- 
crimination step 33 by processing executed by the CPU 
2001 . 

At step S501 , a color histogram is created from the 
pixel values (e.g., the RGB values) of the image seg- 
ment and the histogram is stored in the working memory. 
Next, the number frequency peaks of the color histo- 
gram is detected at step S502. Though the number of 
colors for which the frequency is greater than a prede- 
termined threshold value is adopted as the number of 
peaks, neighboring peaks in the histogram are counted 
as one peak. Next, the variance of the color histogram 
is calculated at step S503, and the image-segment com- 
ponent is discriminated at step S504 from the obtained 
number of peaks and variance. This discrimination in- 
volves judging that the image segment is a limited-color 
image segment if the number of peaks is one and, more- 
over, the variance is less than a predetermined thresh- 
old value. The image segment is judged to be an image 
segment of a continuous color tone in all other cases. 

Next, as shown in Fig. 12, the continuous mono- 
chromatic tone/limited-value discrimination step 34 
makes use of the fact that, in a case where the image 
segment is a limited-value image segment, the pixel val- 
ues of the area concentrate in specific values. Based 
upon this fact, an image segment wherein there is a sin- 
gle lightness at which the frequency in the lightness his- 
togram of the image segment indicates a peak and, 
moreover, there is little variance in the lightness histo- 
gram, is judged to be a limited-value image segment. 

Fig. 16 is a flowchart illustrating a procedure for im- 
plementing the continuous monochromatic tone/limited- 
value discrimination step 34 by processing executed by 
the CPU 2001. 

At step S601 , lightness is calculated from the pixel 
values (e.g., RGB values) in the image segment, a light- 
ness histogram is created and the histogram is stored 
in the working memory. In a case where this is ex- 
pressed in a colorimetric system which already includes 
lightness, as in the case of an HSI colorimetric system, 
the lightness histogram can be created directly. Next, 
the number frequency peaks of the lightness histogram 
is detected at step S602. Though the number of light- 
nesses for which the frequency is greater than a prede- 
termined threshold value is adopted as the number of 
peaks, neighboring peaks in the histogram are counted 
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as one peak. Next, the variance of the lightness histo- 
gram is calculated at step S603, and the image-segment 
component is discriminated at step S604 from the ob- 
tained number of peaks and variance. This discrimina- 
tion involves judging that the image segment is a limited- 5 
value image segment if the number of peaks is one and, 
moreover, the variance is less than a predetermined 
threshold value. The image segment is judged to be an 
image segment of a continuous monochromatic tone in 
all other cases. 10 

The character or line-drawing/pseudo-halftone dis- 
crimination step 35 will now be described. 

Many techniques for discriminating between char- 
acters or line drawings and pseudo-halftone image seg- 
ments have been proposed as image segmentation 1$ 
techniques applied to binary images or multivalued im- 
ages. Here such a technique proposed by the present 
applicant will be described as one example: 

In this embodiment, as set forth above, a binary im- 
age indicating the shape of an input image segment is zo 
crated at the intermediate image segmentation step 3 
and the binary image is stored as image-segment data 
of the form illustrated in Figs. 5A and5B. In the character 
or line-drawing/pseudo-halftone discrimination step 35, 
the image-segment shape (binary image) of an image 2S 
segment to be discriminated is inputted and it is deter- 
mined whether this is the image segment of a character 
or line drawing or the image segment of a pseudo-half- 
tone. 

Fig. 1 7 is a block diagram showing the construction 30 
of an image segmentation unit 3000 for implementing 
the discrimination processing described above. Though 
the image segmentation unit in Fig. 1 7 is implemented 
by its own hardware, it goes without saying that it can 
be implemented also by software and the CPU 2001 of 3$ 
the apparatus shown in Fig. 4. 

As shown in Fig. 17, numeral 3010 denotes a data 
holding unit for sequentially updating and holding each 
pixel value in a small area necessary for image-segment 
discrimination processing. The data holding unit 3010 40 
receives digital binary image data 3040 that is inputted 
in a raster scanning format. As shown in Fig. 18, the 
data holding unit 3010 holds data of a total of 64 pixels 
in an area 324 of a maximum of 8 X 8 pixels neighboring 
a pixel of interest 321 . The data held is updated to data 45 
of the corresponding area in. sync with the updating (se- 
quential movement by raster scanning) of the position 
of the pixel of interest 321 . 

Fig. 1 9 is a block diagram showing the construction 
of the data holding unit 3010. Numeral 3011 denotes a so 
line buffer, which is constituted by a FIFO memories or 
the like, for holding data equivalent to seven lines im- 
mediately preceding the line currently being inputted. 
Numeral 3012 denotes a latch group for holding eight 
pixels of data per line, for a total of 64 items of data. ss 

With reference again to Fig. 17, numeral 3020 de- 
notes an image-segment discriminator for discriminat- 
ing whether the pixel of interest 321 is a pixel in a pseu- 



do-halftone area based upon data of a pixel group which 
includes the pixel of interest 321 forming the small area 
324 outputted by the data holding unit 3010. The image- 
segment discriminator 3020 includes an isolated-pixel 
discriminator 3021 , a periodicity discriminator 3022 and 
a high-frequency component discriminator 3023. These 
three discriminators judge whether conditions that differ 
from one another are satisfied. If any one of the condi- 
tions is satisfied, an OR gate 3024 outputs a signal 3050 
indicating that the pixel of interest 321 is a pixel in a 
pseudo-halftone area. More specifically, the OR gate 
3024 outputs the signal 3050, which is indicative of the 
pseudo-halftone area, if the pixel of interest 321 is an 
isolated point or a pixel contained in an isolated-pixel 
area, a periodic-pixel area or a high-frequency compo- 
nent area. 

The isolated-pixel discriminator 3021 determines 
whether the pixel of interest 321 resides in an isolated- 
pixel area. More specifically, with regard to each of 16 
pixels in a 4 x 4 pixel area 322 shown in Fig. 20, it is 
determined by detection whether four pixels (Fig. 21), 
namely the pixels above, below and to the left and right, 
are all pixels of identical value (all white pixels or all 
black pixels) and whether the values of these four pixels 
differ from the value of the center pixel (k.l). If the results 
of detecting the 1 6 pixels are obtained and two or more 
of these pixels satisfy the above-mentioned condition, 
then it is decided that the pixel of interest 321 in the 4 
x 4 pixel area 322 resides in an isolated-pixel area. In 
other words, among the 16 pixels in the pixel area 322 
neighboring the pixel of interest 321 , it is determined 
whether isolated pixels (black pixels or white pixels not 
connected at the top. bottom, left or right) are two or 
more in number. If isolated pixels are two or more, then 
the pixel of interest 321 is judged to reside in an isolated- 
pixel area. 

The periodicity discriminator 3022 determines 
whether a pixel value is repeated at a predetermined 
pixel period. More specifically, with the exception of a 
case in which the 64 pixels in the 8 x 8 pixel area 324 
in Fig. 18 are all white pixels or all black pixels, it is de- 
termined whether pixel values of four pixels segmented 
from each other by four pixels in the main-scan direction 
and/or by four pixels in the sub-scan direction, e.g., the 
pixel values of pixels A, B, C, D indicated by the hatching 
in Fig. 18, are all the same or not. It should be noted 
that, in the 8 x 8 pixel area 324, It Is possible to define 
16 sets of four pixels in the relative positional relation- 
ship of the kind indicated by the pixels A, B, C, D. In 
each of these 1 6 sets, it is determined whether the pixel 
values of the four pixels are all equal or not. If they are, 
say, 1 4 sets in which the four pixels are all equal, it is 
determine that the portion is a periodic portion, namely 
a dither portion, with the exception of a case where all 
64 pixels are white pixels or black pixels. 

The high-frequency component discriminator 3023 
determines whether a change is density occurs fre- 
quently between adjacent pixels. More specifically, in a 
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6x6 pixel area shown in Fig. 22, the discriminator 3023 
determines whether there is a difference between the 
pixel values of two mutually adjacent pixels in 60 sets 
thereof obtained by combining 30 combinations (indicat- 
ed by the horizontal bidirectional angles in Fig. 22) be- 
tween two mutually adjacent pixels of 36 pixels in the 
horizontal (main -scan) direction and 30 combinations 
(indicated by the vertical bidirectional angles in Fig. 22) 
between two mutually adjacent pixels of 36 pixels in the 
vertical (sub-scan) direction. Further, if there are, say, 
28 or more sets of mutually different combinations, the 
discriminator 3023 judges that a pixel resides in a high- 
frequency component area. 

The results of discrimination (the image-segment 
components of each image segment) at the image-seg- 
ment discrimination step 3 are stored, as image-seg- 
ment components of each image segment, in the image- 
segment data in the image-segment data memory 1007. 
With regard to an image segment judged as being each 
image-segment component of a limited-color character 
or line drawing, a limited-color pseudo-halftone, a limit- 
ed-value character or line drawing or a limited-value 
pseudo-halftone, the mean color in the image segment 
(e.g. , the average pixel value of each of the colors RGB) 
is calculated and this is stored, as the mean color of the 
image segment, in the image-segment data in the im- 
age-segment data memory 1 007. 

Adaptive zooming step 

The adaptive zooming step 4 subjects each image 
segment extracted at the image-segment extraction 
step 2 to zoom processing conforming to the image-seg- 
ment component discriminated at the image-segment 
discrimination step 3, thereby creating a zoomed image 
of the input image. 

Fig. 23 is a diagram illustrating the detailed proce- 
dure of the adaptive zooming step 4. The adaptive 
zooming step 4 adapts the zooming step, which con- 
forms to the image-segment component, for every im- 
age segment and then combines the zoomed images of 
the respective image segments, thereby creating a 
zoomed image of the input image. This is performed un- 
der the control of a zoom control step 41 . More specifi- 
cally, the image segment of a continuous color tone is 
zoomed at a continuous color-tone image-segment 
zooming step 42, the image segment of a limited-color 
character or line drawing is zoomed at a limited-color 
character/line-drawing image-segment zooming step 
43, the image segment of a limited-color pseudo-half- 
tone is zoomed at limited-color pseudo-halftone image- 
segment zooming step 44, the image segment of a con- 
tinuous monochromatic tone is zoomed at a continuous 
monochromatic-tone image-segment zooming step 45, 
the image segment of a limited-value character or line 
drawing is zoomed at a limited-value character/line- 
drawing image-segment zooming step 46, the image 
segment of a limited-value pseudo-halftone is zoomed 



by a limited-value pseudo-halftone image-segment 
zooming step 47, and the resulting zoomed image seg- 
ments are combined into a single zoomed image. Each 
of these steps will now be described. 

5 The zoom control step 41 adaptively applies the 
zoom process conforming to an image-segment com- 
ponent to each and every image segment until there are 
no longer any image segments for which zooming has 
not been completed. Fig. 24 is a diagram illustrating a 

10 sequence of image segments which undergo zooming. 
Zooming is performed while pursuing image-segment 
nodes, which are described in image-segment data 
(having the tree structure), from route nodes (the most 
significant nodes). However, the zooming of each image 

is segment must not be carried out ahead of a background 
image segment which is one rank below a node superior 
to the image segment itself. In other words, in a case 
where an image segment inferior to one intermediate 
image segment is zoomed, zooming must be performed 

20 from the background image segment. 

The continuous color-tone image-segment zoom- 
ing step 42 zooms an inputted image segment as by a 
color image zooming method using linear interpolation 
and writes the data of the zoomed image in the zoomed- 

25 image memory 1 008. Though the writing of the zoomed- 
image data in the zoomed-image memory 1008 is the 
same as in other steps, this performed based upon the 
position of the zoomed image segment calculated from 
the image-segment position of the inputted image seg- 

30 ment. 

According to the limited-color character/line draw- 
ing image-segment zooming step 43, the binary image 
stored as the image-segment shape data of the inputted 
image segment is zoomed by a well-known binary-im- 
35 age zooming method suited to a character or line-draw- 
ing image, such as the method in Japanese Patent Ap- 
plication No. 

3-345062 already filed by the applicant, and the data of 
this zoomed image is written in the zoomed-image 

40 memory 1008. According to the limited-color pseudo- 
halftone image-segment zooming step 44, the binary 
image stored as the image-segment shape data of the 
inputted image segment is zoomed by a binary-image 
zooming method such as the SPC zooming method and 

45 the data of this zoomed image is written in the zoomed- 
image memory 1008. In the limited-color character/line 
drawing image-segment zooming step 43 and the limit- 
ed-color pseudo-halftone image-segmenl zooming step 
44, the writing of the zoomed image data is performed 

so by writing the pixel value of the input image in a case 
where pixel value of the zoomed binary image is "1 " and 
not writing a pixel for which the pixel value is "0°. A value 
written need not be a pixel value in the input image but 
may be an image-segment color of the inputted image 

55 segment. If such is the case, noise contained in the input 
image can be removed. 

According to the continuous monochromatic-tone 
image-segment zooming step 45, an inputted image 
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segment is zoomed as by a color-image zooming meth- 
od using linear interpolation and the data of the zoomed 
image is written in the zoomed-image memory 1008. 

According to the limited-value character/line draw- 
ing image-segment zooming step 46, the binary image 5 
stored as the image-segment shape data of the inputted 
image segment is zoomed by the well-known binary-im- 
age zooming method suited to a character or line-draw- 
ing image, such as the method in Japanese Patent Ap- 
plication No. 3-345062 already filed by the applicant, io 
and the data of this zoomed image is written in the 
zoomed-image memory 1008. According to the limited- 
value pseudo-halftone image-segment zooming step 
47, the binary image stored as the image-segment 
shape data of the inputted image segment is zoomed is 
by a binary-image zooming method such as the SPC 
zooming method and the data ol this zoomed image is 
written in the zoomed-image memory 1008. In the lim- 
ited-value character/line drawing image-segment 
zooming step 46 and Ihe limited-value pseudo-half lone so 
image-segment zooming step 47, the writing of the 
zoomed image data is performed by writing the pixel val- 
ue of the input image in a case where pixel value of the 
binary image, which is obtained by SPC zooming of the 
binary image stored as the image-segment shape data 25 
of the inputted image segment, is "1 " and not writing a 
pixel for which the pixel value is "0°. A value written need 
not be a pixel value in the input image but may be an 
area color of the inputted image segment or the color 
black (e.g., in a case where the output image consists 30 
of eight bits per each of the colors R, G, B, all pixel val- 
ues of RGB are zero). If such is the case, noise con- 
tained in the input image can be removed. 

Further, the zoom ratio employs a value stored in 
the ROM 2004 in advance or a value designated by the 3S 
user from a keyboard or DIP switch. 

Thus, in accordance with this embodiment as de- 
scribed above, a document image inputted as a color 
image and consisting of a mixture of image segments 
having different components is subjected to image seg- 40 
mentation based upon the features of the color docu- 
ment image, whereby it is possible to segment image 
segments from the document image in excellent fashion 
even if the document image is one having a plurality of 
backgrounds whose colors are other than white. A seg- *s 
mented image segment is subjected to zoom process- 
ing that conforms to the characteristics of this area, as 
a result of which a zoomed color image having excellent 
image quality can be obtained. 

so 

<Second Embodiment 

An image processing apparatus according to a sec- 
ond embodiment of the present invention will now be 
described. Components in the second embodiment sim- 55 
ilar to those of the first embodiment are designated by 
like characters and need not be described in detail 
again. 



[Processing procedure] 

Fig. 25 is a diagram showing a processing proce- 
dure for a case in which the present invention is applied 
to processing for compressing a color image in an image 
processing apparatus. Steps from the color image input 
step 1 to the image-segment discrimination step 3 and 
the color image output step 5 are the same as in the first 
embodiment and a detailed description thereof is omit- 
ted. 

In Fig. 25, an adaptive compression step 6 creates 
compressed data indicative of input image data by ap- 
plying each image segment extracted at the image-seg- 
ment extraction step 2 to compression processing that 
conforms to the image-segment component of the im- 
age segment discriminated by the image-segment dis- 
crimination step 3. According to a compressed-data out- 
put step 7, the compressed data created at the adaptive 
compression step 6 is outputted to a communication line 
or image storage device, which are not shown. The data 
compressed at the adaptive compression step 6 inputs 
from a communication line or image storage device at 
a compressed-data input step 8. The data compressed 
at the adaptive compression step 6 is reproduced as a 
color image at a color image reproduction step 9. It 
should be noted that the steps from the color image input 
step 1 to the compressed-data output step 7 are exe- 
cuted when the color image data is compressed. The 
steps from the compressed-data input step 8 to the color 
image output step 5 are executed when the compressed 
data is reproduced. 

[Construction of apparatus] 

Fig. 26 is a block diagram showing the construction 
of an image processing apparatus which executes the 
processing shown in Fig. 25, and Fig. 27 is a block dia- 
gram showing hardware for implementing the arrange- 
ment of Fig. 26. 

As shown in Fig. 27, this embodiment is further pro- 
vided with a data input unit 2010 and a data output unit 
201 1 , both of which are connected to the I/O port 2005. 

As shown in Fig. 26, the color image input unit 1001 
inputs digital color image data to be processed and 
stores this data in the input image memory 1006. The 
image-segment extraction unit 1002 extracts image 
segments from the image that has been stored in an in- 
put image memory 1006 and stores the extracted im- 
age-segment data in an image-segment data memory 
1007. The image-segment discrimination unit 1003 dis- 
criminates image-segment components in each image 
segment extracted by the image-segment extraction 
unit 1 002 and stores the image-segment components in 
the image-segment data memory 1007. An adaptive 
compression unit 1014 creates compressed data indic- 
ative of input image data by applying each image seg- 
ment extracted by the image-segment extraction unit 
1002 to compression processing that conforms to the 
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image-segment component of the image segment dis- 
criminated by the image-segment discrimination unit 
1003. The compressed data is stored in a compressed- 
data memory 1012. 

A compressed-data output unit 1009 outputs com- 
pressed data compressed by the adaptive compression 
unit 1014 and stored in the compressed-data memory 
1 01 2. The destination of this output is a communication 
line or an image storage device. An arrangement can 
be adopted in which, under the control of the CPU 2001 
which operates in accordance with a program stored in 
the ROM 2004 or storage device 2002. the data output 
unit 2011 outputs the compressed data that has been 
stored In the RAM 2006 or storage device 2002. 

The compressed-data input unit 1011 inputs the 
compressed data from the communication line or image 
storage device and stores the compressed data in the 
compressed-data memory 1012. An arrangement can 
be adopted in which, under the control of the CPU 2001 
which operates in accordance with the program stored 
in the ROM 2004 or storage device 2002, compressed 
data sent via the communication line or compressed da- 
ta stored in the image storage device is inputted by the 
data input unit 2010. 

A color image reproduction unit 1010 reproduces 
color image data from the compressed data that has 
been stored in the compressed-data memory 1012 and 
stores the color image data in a reproduced-image 
memory 101 3. The reproduced color image data stored 
in the reproduced-image memory 1013 is outputted by 
the color image output unit 1 005. The destination of this 
output is a display, a printer or copier for making hard 
copies, a communication line or an image storage de- 
vice. 

It should be noted that the adaptive compression 
unit 1014 and color image reproduction unit 1010 can 
be realized by the CPU 2001 , which operates in accord- 
ance with the program stored in the ROM 2006 or stor- 
age device 2002, and the RAM 2006 or memory device 
2002 used as a working memory. It goes without saying 
that these can also be realized by their own CPUs, 
RAMs and memories or by their own hardware. Further, 
the compressed-data memory 1012 and the repro- 
duced-image memory 101 3 can be realized by the RAM 
2006 or memory device 2002. It goes without saying that 
these memories can be realized by their own storage 
devices. 

Furthermore, in a case where compression of the 
data and reproduction of the data are carried out in seg- 
ment units, as when compressed data is sent and re- 
ceived via a communication line, this can be realized 
using two devices, such as a device which performs da- 
ta compression and a device which reproduces com- 
pressed data, as illustrated in Fig. 28. 

[Form of compressed data] 

The form of the compressed data will be described 



next. A tree structure in which the image segments 
shown in Fig. 5A serve as nodes is retained also after 
the image data has been compressed. 

Fig. 29 is a diagram showing the structure of the 

5 compressed data. The compressed data is composed 
of a header (referred to as a "compressed-data header' 
beiow), in which are registered the image size and the 
compression method for the type of image-segment 
component, and the compressed data (referred to as 

10 "compressed image-segment data" below) of each im- 
age segment. The compressed image-segment data 
contains the image-segment data of the image seg- 
ment. 

Figs. 30A and 30B are diagrams showing the form 

is of image-segment data contained in compressed im- 
age-segment data. Image-segment data whose form is 
either of two types is held in dependence upon the im- 
age-segment component. More specifically, in case of 
a "limited-color character or line-drawing image seg- 

20 ment", a "limited-color pseudo-halftone image seg- 
ment", a "limited-value character or line-drawing image 
segment" or a "limited-value pseudo-halftone image 
segment" , image-segment data of the form shown in 
Fig. 30A is held. Each element shown in Fig. 30A cor- 

25 responds to a respective ones of the elements shown 
in Fig. 5B. In the first embodiment, a binary image indic- 
ative of the shape of an image segment is registered as 
"image-segment shape". In the second embodiment, 
however, a binary image compressed by a compression 

30 method stored in the compressed-data header is regis- 
tered as "image-segment shape". 

In the case of a "continuous color-tone image seg- 
ment" or a "continuous monochromatic-tone image seg- 
ment", image-segment data of the form shown in Fig. 

35 30B is held. It should be noted that elements other than 
"image-segment shape" and "image-segment image 
data" correspond to respective ones of the elements 
shown in Fig. SB, data compressed by a compression 
method registered in the compressed-data header is 

40 registered as "image-segment image data" and a binary 
image compressed by a method of compressing a lim- 
ited-value character or line-drawing image segment is 
registered as "image-segment shape". 

45 [Details of processing] 

The main steps shown in Fig. 25 will now be de- 
scribed in detail while using the arrangement of Fig. 26. 
However, steps from the color image input step 1 to the 
so image-segment discrimination step 3 and the color im- 
age output step 5 are the same as in the first embodi- 
ment and a detailed description thereof is omitted. 

Adaptive compression step 

55 

The adaptive compression step 6 creates com- 
pressed data indicative of input image data by applying 
each image segment extracted at the image-segment 
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extraction step 2 to compression processing that con- 
forms to the image-segment component discriminated 
by the image-segment discrimination step 3. 

Fig. 31 is a diagram showing the detailed procedure 
of the adaptive compression step 6. In accordance with s 
a compression control step 61 , the adaptive compres- 
sion step 6 adapts the compression step, which con- 
forms to the particular image-segment component, for 
every image segment, thereby creating compressed da- 
ta for each image segment, and stores the compressed io 
data in the compressed-data memory 1012. More spe- 
cifically, the image segment of a continuous color tone 
is compressed at a continuous color-tone image-seg- 
ment compression step 62, the image segment of a lim- 
ited-color character or line drawing is compressed at a is 
limited-color character/line drawing image-segment 
compression step 63, the image segment of a limited- 
color pseudo-halftone is compressed at a limited-color 
pseudo-halftone image-segment compression step 64, 
the image segment of a continuous monochromatic tone 20 
is compressed at a continuous monochromatic-tone im- 
age-segment compression step 65, the image segment 
of a limited-value character or line drawing is com- 
pressed at a limited-value character/line-drawing im- 
age-segment compression step 66, and the image seg- 2S 
ment of a limited-value pseudo-halftone is compressed 
by a limited-value pseudo-halftone image-segment 
compression step 67. Each of these steps will now be 
described. 

The compression control step 61 adapt ively applies 30 
the compression process conforming to an image-seg- 
ment component to each and every image segment until 
there are no longer any image segments that have not 
been compressed. 

The continuous color-tone image-segment com- 3S 
pression step 62 compresses a continuous color-tone 
image segment and stores the created compression da- 
ta in the compressed-data memory 1012. More specifi- 
cally, an image portion corresponding to the image-seg- 
ment position of node to be processed is compressed 40 
by a coding method suited to a continuous color tone 
and the compressed data is registered as the "image- 
segment image data" shown in Fig. 30B. Further, the 
"image-segment shape" in the image-segment data 
stored in the image-segment data memory 1 007 is com- 45 
pressed by the compression method of a limited-value 
character or line-drawing image segment, the com- 
pressed data is registered in the "image-segment 
shape" shown in Fig. 30B, and values the same as the 
image-segment data are registered as the other ele- so 
ments. It should be noted that JPEG coding using DCT, 
for example, is utilized as the coding method suited to 
the continuous color tone. 

The limited-color character/line drawing image- 
segment compression step 63 compresses a limited- ss 
color character or line-drawing image segment and 
stores the created compressed data in the compressed- 
data memory 1012. More specifically, a binary image 



stored as the image-segment shape ol node to be proc- 
essed is compressed by coding suited to a limited-color 
character or line drawing, the compressed data is reg- 
istered as "image-segment shape" shown in Fig. 30A. 
and values the same as the image-segment data are 
registered as the other elements. It should be noted that 
MR/MMR coding, for example, is utilized as the coding 
method suited to the limited-color character or line draw- 
ing. 

The limited-color pseudo-halftone image-segment 
compression step 64 compresses a limited-color pseu- 
do-halftone image segment and stores the created com- 
pressed data in the compressed-data memory 1012. 
More specifically, a binary image stored as the image- 
segment shape of node to be processed is compressed 
by coding suited to a limited-color pseudo-halftone, the 
compressed data is registered as "image-segment 
shape" shown in Fig. 30A, and values the same as the 
image-segment data are registered as the other ele- 
ments. It should be noted that JBIG coding, for example, 
is utilized as the coding method suited to the limited- 
color pseudo-halftone. 

The continuous monochromatic-tone image-seg- 
ment compression step 65 compresses a continuous 
monochromatic-tone image segment and stores the 
created compressed data in the compressed-data 
memory 1012. More specifically, an image portion cor- 
responding to the image-segment position of a node to 
be processed is compressed by coding suited to a con- 
tinuous monochromatic tone and the compressed data 
is registered as "image-segment image data" shown in 
Fig. 30B. Further, "image-segment shape" in the image- 
segment data stored in the image-segment data mem- 
ory 1007 is compressed by a compression method for 
a limited-value character or line-drawing image seg- 
ment and the compressed data is registered as "image- 
segment shape" shown in Fig. 30B, and values the 
same as the image-segment data are registered as the 
other elements. It should be noted that JPEG coding, 
for example, is utilized as the coding method suited to 
the continuous monochromatic tone. 

The limited-value character/line drawing image- 
segment compression step 66 compresses a limited- 
value character or line-drawing image segment and 
stores the created compressed data in the compressed- 
data memory 1012. More specifically, a binary image 
stored as the image-segment shape ol node to be proc- 
essed is compressed by coding suited to a limited-value 
character or iine drawing, the compressed data is reg- 
istered as "image-segment shape" shown in Fig. 30A, 
and values the same as the image-segment data are 
registered as the other elements. It should be noted that 
MR/MMR coding, for example, is utilized as the coding 
method suited to the limited-value character or line 
drawing. 

The limited-value pseudo-halftone image-segment 
compression step 67 compresses a limited-value pseu- 
do-halftone image segment and stores the created com- 
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pressed data in the compressed-data memory 1012. 
More specifically, a binary image stored as the image- 
segment shape of node to be processed is compressed 
by coding suited to a limited-value pseudo-halftone, the 
compressed data is registered as "image-segment 
shape" shown in Fig. 30A, and values the same as the 
image-segment data are registered as the other ele- 
ments. It should be noted that JBIG coding, for example, 
is utilized as the coding method suited to the limited- 
vatue pseudo-halftone. 

Color image reproduction step 

The color image reproduction step 9 reproduces 
color image data from compressed data that has been 
stored in the compressed-data memory 1012. 

Fig. 32 is a diagram showing the detailed procedure 
of the color image reproduction step 9. In accordance 
with a reproduction control step 91 , the color image re- 
production step 9 reproduces a color image by applying, 
for each image segment, a reproduction process con- 
forming to the "image-segment type" of the image-seg- 
ment data contained in the compressed image-segment 
data, and stores this color image is stored the repro- 
duced-image memory 1013. More specifically, repro- 
duction is performed at a continuous color-tone image- 
segment reproduction step 92 in case of a continuous 
color tone, at a limited-color character/line drawing im- 
age-segment reproduction step 93 in case of a limited- 
color character or line drawing; at a limited-color pseu- 
do-halftone image-segment reproduction step 94 in 
case of a limited-color pseudo-halftone; at a continuous 
monochromatic-tone image-segment reproduction step 
95 in case of a continuous monochromatic tone; at a 
limited-value character/line -drawing image-segment re- 
production step 96 in case of a limited-value character 
or line drawing; and at a limited-value pseudo-halftone 
image-segment reproduction step 97 in case of a limit- 
ed-value pseudo-halftone. The reproduced image seg- 
ments are combined into a single reproduced image. 
Each of these steps will now be described. 

The reproduction control step 91 adaptiveiy applies 
the reproduction process conforming to the image-seg- 
ment type to each and every image segment until there 
is no longer any compressed image-segment data that 
has not been reproduced. The reproduction of image 
segments is performed in the sequence shown in Fig. 
24. In other words, image-segment reproduction is per- 
formed while pursuing image-segment nodes, which are 
described in image-segment data (having the tree struc- 
ture), from route nodes (the most significant nodes). 
However, the reproduction of each image segment must 
not be carried out ahead of a background image seg- 
ment which is one rank below a node superior to the 
image segment itself. In other words, in a case where 
an image segment inferior to one intermediate image 
segment is reproduced, reproduction must be per- 
formed from the background image segment. 



The continuous color-tone image-segment repro- 
duction step 92 reproduces an image segment from 
compressed image-segment data of a continuous color 
tone and stores this image segment in the reproduced- 

5 image memory 1013. Fig. 33 is a flowchart illustrating 
the detailed procedure of the continuous color-tone im- 
age-segment reproduction step 92. This flowchart is ex- 
ecuted by the CPU 2001 , by way of example. 

First at step S701 data registered as "image-seg- 

io ment shape" of the compressed image-segment data is 
decoded by a decoding method corresponding to the 
method of coding of the limited-value character or line- 
drawing image segment registered in the compressed- 
data header. As a result, binary image data indicating 

is the image-segment shape is reproduced and stored in 
the working memory. At step S702, data registered as 
"image-segment image data" of the compressed image- 
segment data is decoded by a decoding method corre- 
sponding to the method of coding of the continuous 

20 color-tone image segment registered in the com- 
pressed-data header. As a result, image-segment im- 
age data is reproduced and stored in the working mem- 
ory. At step 703, pixels of the reproduced image-seg- 
ment image data which correspond to pixels of value "1 " 

25 jn the reproduced binary -image data are stored in the 
reproduced-image memory 1013. It should be noted 
that the writing of reproduced image data in the repro- 
duced-image memory 1013 is the same as in the other 
steps and is performed based upon position calculated 

30 from "image-segment position" of the compressed im- 
age-segment data. 

The limited-color character/line drawing image- 
segment reproduction step 93 reproduces an image 
segment from compressed image-segment data of a 

35 limited-color character or line drawing and stores the im- 
age segment in the reproduced-image memory 1013. 
Fig. 34 is a flowchart illustrating the detailed procedure 
of the limited-color character/line drawing image-seg- 
ment reproduction step 93. This flowchart is executed 

40 by the CPU 2001 , by way of example. 

, First at step S801 , data registered as "image-seg- 
ment shape" of the compressed image-segment data is 
decoded by a decoding method corresponding to the 
method of coding of the limited-color character or line- 

45 drawing image segment registered in the compressed- 
data header. As a result, binary image data indicating 
the image-segment shape is reproduced and stored in 
the working memory. Next, at step S802, pixels regis- 
tered as the "image-segment color" of the compressed 

so image-segment data are stored at positions in the re- 
produced-image memory 1 01 3 which correspond to pix- 
els of value "1 " in the reproduced binary-image data. 

The limited-color pseudo-halftone image-segment 
reproduction step 94 reproduces an image segment 

55 from compressed image-segment data of a limited-color 
pseudo-halftone and stores the image segment in the 
reproduced-image memory 1013. The limited-color 
pseudo-halftone image-segment reproduction step 94 
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reproduces image segments through a procedure sim- 
ilar to that of the limited-color character/line drawing im- 
age-segment reproduction step 93 illustrated in Fig. 34. 

The continuous monochromatic-tone image-seg- 
ment reproduction step 95 reproduces an image seg- 
ment from compressed image-segment data of a con- 
tinuous monochromatic tone and stores this image seg- 
ment in the reproduced-image memory 1013. The con- 
tinuous monochromatic-tone image-segment reproduc- 
tion step 95 reproduces image segments through a pro- 
cedure similar to that of the continuous color-tone im- 
age-segment reproduction step 92 shown in Fig. 33. 

The limited-value character/line drawing image- 
segment reproduction step 96 reproduces an image 
segment from compressed image-segment data of a 
limited-value character or line drawing and stores the 
image segment in the reproduced-image memory 1 01 3. 
The limited-value pseudo-halftone image-segment re- 
production step 97 reproduces an image segment from 
compressed image -segment data of a limited-value 
pseudo-halftone and stores the image segment in the 
reproduced-image memory 1013. The limited-value 
character/line drawing image-segment reproduction 
step 96 and the limited-value pseudo-halftone image- 
segment reproduction step 97 reproduce image seg- 
ments through a procedure similar to that of the limited- 
color character/line drawing image-segment reproduc- 
tion step 93 shown in Fig. 34. 

Thus, in accordance with this embodiment as de- 
scribed above, a document image inputted as a color 
image and consisting of a mixture of image segments 
having different components is subjected to image seg- 
mentation based upon the features of the color docu- 
ment image, whereby it is possible to segment image 
segments from the document image in excellent fashion 
even if the document image is one having a plurality of 
backgrounds whose colors are other than white. A seg- 
mented image segment is subjected to compression 
processing that conforms to the characteristics of this 
area, as a result of which compressed data can be ob- 
tained at an excellent compression rate. In addition, 
compressed image-segment data contained in this 
compressed data can be decompressed by a method 
conforming to the characteristics of the image segment, 
thereby making it possible to obtain a reproduced color 
image having an excellent image quality. 

<Third Embodiment 

An image processing apparatus according to a third 
embodiment of the present invention will now be de- 
scribed. Components in the third embodiment similar to 
those of the first embodiment are designated by like 
characters and need not be described in detail again. 

Next, in order to input a color original image into a 
DTP apparatus in the form of structured data, an em- 
bodiment will be described in which the present inven- 
tion is applied to an image processing apparatus which 



edits a document image. 

Fig. 35 is a diagram showing a processing proce- 
dure for a case in which the present invention is applied 
to document image editing processing in an image 
5 processing apparatus. Steps from the color image input 
step 1 to the image-segment discrimination step 3 are 
the same as those of the first embodiment and need not 
be described in detail again. 

According to a data conversion step 10, image-seg- 

io ment data created by the process up to the image^seg- 
ment discrimination step is converted, for each image 
segment, to image data having a data format suitable 
for the editing of a document image in a DTP apparatus 
or the like. As a result, an input image can be applied to 

is document-image editing processing per each image 
segment segmented by the process up to the image- 
segment discrimination step 3, and this editing can also 
be performed per each image segment. 

In the data conversion step 10, characters or sym- 

20 bols contained in a limited-color character of line-draw- 
ing image segment or limited-value character of line- 
drawing image segment can also be converted to char- 
acter codes by character recognition in dependence up- 
on a designation made by the user. Further, with regard 

2S to a limited-color character of line-drawing image seg- 
ment or limited-value character of line-drawing image 
segment, data having a raster-scanning format can also 
be converted to vector data by raster-vector conversion 
in dependence upon a designation made by the user. 

30 in a document image editing step 11, a character 
image is edited in accordance with a designation from 
the user and a document image representing the results 
of editing is outputted. 

The foregoing processing can also be implemented 

35 in a DTP apparatus or in a special-purpose apparatus 
illustrated in the first embodiment. 

Thus, in accordance with this embodiment, a doc- 
ument image inputted as a color image and consisting 
of a mixture of image segments having different compo- 

40 nents is subjected to image segmentation based upon 
the features of the color document image, whereby it is 
possible to segment image segments from the docu- 
ment image in excellent fashion even if the document 
image is one having a plurality of backgrounds whose 

45 colors are other than white. A color original image can 
be structured and inputted into a document-image edit- 
ing apparatus such as a DTP apparatus. In addition, a 
segmented image segment is subjected to editing 
processing conforming to the characteristics of the im- 

so age segment so that a color document image having ex- 
cellent image quality can be obtained. 

<Fourth Embodiment 

55 An image processing apparatus according to a 
fourth embodiment of the present invention will now be 
described. Components in the fourth embodiment sim- 
ilar to those of the first embodiment are designated by 
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like characters and need not be described in detail 
again. 

In the first embodiment, a background image seg- 
mentation step determines whether an intermediate im- 
age segment contains a background image segment 
and then segments the background image segment. In 
the example described, this step entails extracting, as 
the background image segment, an image segment of 
the same color occupying the major part of the interme- 
diate image segment to be processed. Such method is 
effective in many color document images. However, in 
a color document image containing a continuous tone 
image segment such as a photographic image having 
portions in which there is little change in brightness or 
color, there are occasions where a background image 
segment is judged to reside in a continuous tone image 
segment that must properly be judged as being a single 
image segment, with the result that the continuous tone 
image segment is needlessly segmented into a contin- 
uous tone image segment and an intermediate image 
segment. 

Accordingly, in this embodiment, a background im- 
age segmentation process is described in which, by 
judging whether an extracted background image seg- 
ment is appropriate or not, it is possible to prevent the 
needless segmenting of a continuous tone image seg- 
ment having a portion in which there is little change in 
brightness or color. 

Fig. 36 is a diagram showing the detailed procedure 
of the background image segmentation step 22 accord- 
ing to this embodiment. This corresponds to Fig. 7 of the 
first embodiment. 

In Fig. 36, a background color extraction step 221 
analyzes the colors of the inputted intermediate image 
segment and extracts the color of the background image 
segment. More specifically, in the inputted intermediate 
image segment, a pixel having the most prevalent color 
or a color close to this color in comparison with the pixels 
of other colors has this most prevalent color extracted. 
A background image-segment extraction step 222 uses 
the extracted color of the background image segment 
to create a binary image of the background image seg- 
ment. A background image-segment discrimination step 
223 judges whether an image segment indicated by the 
created binary image is appropriate as a background im- 
age. If the image segment indicated is judged to be ap- 
propriate, processing proceeds to a background image- 
segment shaping step 224. If the image segment indi- 
cated is judged to be inappropriate, processing is termi- 
nated on the grounds that there is no background image 
segment or processing returns to the background im- 
age-segment extraction step 222, the processing pa- 
rameters are changed and a background image seg- 
ment is extracted again. The background image-seg- 
ment shaping step 224 removes noise and small image 
segments from the binary image indicating the image 
segment judged as being an appropriate background 
image segment, shapes the background image seg- 



ment and verifies, by the processing for removing small 
areas, whether the background image segment will van- 
ish. 

Figs. 37A through 37C are diagrams for describing 

s an overview of processing for judging a background im- 
age segment. Fig. 37A shows an example in which a 
background image segment is extracted from an inter- 
mediate image segment containing a background im- 
age segment, Fig. 37B shows an example in which a 

io background image segment is extracted from an inter- 
mediate image segment of a continuous tone image 
segment such as a photograph, and Fig. 37C shows an 
example in which a background image segment is ex- 
tracted from an intermediate image segment of a color 

is document image having a large amount of noise. The 
background image-segment discrimination step 223 
judges that the result of extraction shown in Fig. 37 A is 
indicative of an appropriate background image segment 
and judges that the results of extraction shown in Figs. 

20 37B and 37C are indicative of inappropriate background 
image segments. In other words, in a case where the 
peripheral shape of an extracted image segment has 
been segmented from the peripheral portion of an input- 
ted intermediate image segment, as shown in Fig. 37B, 

25 the background image- image segment discrimination 
step 223 determines the existence of a continuous tone 
portion having a small change in brightness or color and 
judges that the extracted image segment is inappropri- 
ate as a background image. In a case where there are 

30 many image segments of small area (these are mean- 
ingless as intermediate image segments) located in a 
portion contained in a newly generated intermediate im- 
age segment, namely in an image segment extracted 
as a background image segment, as shown in Fig. 37C, 

35 the range of color extracted as the background image 
segment is narrow. Consequently, the background im- 
age-image segment discrimination step 223 determines 
that a background image segment has not been extract- 
ed sufficiently and judges that the extracted image seg- 

40 ment is inappropriate as a background image. 

Fig. 33 is a diagram illustrating the detailed proce- 
dure of the background image-segment discrimination 
step 223. A background image-segment shape discrim- 
ination step 2231 determines whether the shape of an 

45 image segment extracted as a background image seg- 
ment is broad enough in the inputted intermediate image 
segment. A generated intermediate image-segment dis- 
crimination step 2232 determines whether a large 
number of image segments of small area reside in a 

so newly generated intermediate image segment, as de- 
scribed above. 

Figs. 39 through 41 show a flowchart illustrating a 
procedure for implementing the background image seg- 
mentation step of this embodiment by processing exe- 

55 cuted by the CPU 2001 . This corresponds to Fig. 8 of 
the first embodiment. 

Steps S901 and S902 correspond to the back- 
ground color extraction step 221 . A color histogram is 
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created in the working memory at step S901 , and a color 
having the maximum frequency of occurrence (namely 
a candidate for the background color) is obtained at step 
S902. 

Step S903. which corresponds to the background 
image-segment extraction step 222, creates binary im- 
age data in the working memory. This is binary image 
data of an intermediate image segment that has been 
binarized to pixels having a color close to the color of 
maximum frequency of occurrence and pixels other than 
these pixels. For example, binary image data is created 
in which pixels having a color close to the color of max- 
imum frequency of occurrence are made "1 °s and pixels 
other than these pixels are made B 0 B s. Whether or not 
a pixel of interest has a color near the color of maximum 
frequency is determined by obtaining the distance D in 
accordance with the equation shown below. In other 
words, it is judged that the pixel of interest has a color 
near the color of maximum frequency if the distance D 
obtained falls within a predetermined range (referred to 
as a "background range" below). 

D 2 = (RO - R1 ) 2 + (GO - G1 ) 2 + (BO - B1 ) 2 
where RO, GO, BO are R, G, B values of maximum fre- 
quency of appearance, and R1 , G1 , B1 are R, G, B val- 
ues of the pixel of interest. 

The value of the background range uses data 
stored in the ROM 2004 or the like as the initial value 
thereof but the value is changed in dependence upon 
the result of discrimination performed at the background 
image-segment discrimination step 223. 

The steps from step S904 to S908 correspond to 
the background image-segment shape discrimination 
step 2231. The area (which shall be referred to as the 
"background outside area") on the outer side of the im- 
age segment extracted as the background image seg- 
ment is measured at step S904, and the area of the in- 
putted intermediate image segment is measured at step 
S905. 

Fig. 42 is a diagram for describing a method of 
measuring the background outside area. First, with re- 
gard to each sub-scan of an inputted intermediate image 
segment, the number of pixels (representative of area) 
for which flags have not been raised (the pixels corre- 
spond to the arrows in Fig. 42) is measured from left and 
right boundaries of the Intermediate image segment to 
the sides of the image segment extracted as the back- 
ground image segment, and the flags are raised for the 
pixels thus measured. Next, a similar measurement is 
performed with regard to the main scan but pixels whose 
flags have raised by the measurement in the sub-scan 
direction are not measured. 

Next, at step S906, the ratio of the measured back- 
ground outside area to the area of the intermediate im- 
age segment [= (measured background outside area)/ 
(area of intermediate image segment), hereinafter re- 
ferred to as the "background outside-area ratio"] is ob- 
tained and this ratio is compared with a predetermined 



threshold value. When the ratio is less than the thresh- 
old value, it is judged that the extracted image segment 
is appropriate as the background image segment and 
the program proceeds to step S909; otherwise, it is 

5 judged that the extracted image segment is inappropri- 
ate as the background image segment and the program 
proceeds to step S907. 

Steps S907 and S908 determine whether extraction 
of a background image segment is to be performed 

io again. This determination uses a re-extraction flag A, 
which indicates whether this is the initial discrimination 
of a background image-segment shape, and the back- 
ground outside area ratio prevailing immediately prior to 
re-extraction (this ratio shall be referred to as the "pre- 

15 ceding extraction ratio"). It should be noted that these 
values are stored in the RAM 2006. If the re-extraction 
flag A is discriminated and this is found to be the initial 
discrimination at step S907, then the program proceeds 
to step S903 and re-extraction is executed; otherwise, 

20 the program proceeds to step S908, where the value of 
(background outside-area rat io)/(p receding extraction 
ratio) is compared with a predetermined threshold val- 
ue. If the threshold value is larger, the program returns 
to step S903 and re-extraction is executed; otherwise, 

2S it is judged that there is no background image segment 
and processing is terminated. It should be noted that in 
the case where re-extraction is performed, the back- 
ground color range is widened. If the discrimination of 
the background image segment of the inputted interme- 

30 diate image segment is finished, the re-extraction flag 
A and the background color range are reset. 

Steps from step S909 to S915 correspond to the 
generated intermediate image-segment discrimination 
step 2232. The binary image representing background 

35 image segment undergoes a black-white reversal at 
step S909, connected components are extracted from 
the black-white reversed binary image at step S910, the 
area of the extracted connected components is meas- 
ured at step S911 , and the number of connected com- 

40 ponents for which the area is less than a predetermined 
threshold value (these connected components shall be 
referred to as "small connected components") is meas- 
ured. The number of small connected components is 
compared with a predetermined threshold value at step 

<5 S913. If the number is less than the threshold value, it 
is judged that the extracted Image segment is appropri- 
ate as the background image segment and the program 
proceeds to step S916; otherwise, it is judged that the 
extracted image segment is inappropriate as the back- 

50 ground image segment and the program proceeds to 
step S914. 

Whether or not extraction of background image 
segment is to be performed again is determined at steps 
S91 4 and S91 5. This determination uses a re-extraction 
55 ftag B, which indicates whether this is the initial discrim- 
ination of a generated intermediate image-segment, 
and the number of small connected components prevail- 
ing immediately prior to re-extraction (this number shall 
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be referred to as the "preceding extraction number"). It 
should be noted that these values are stored in the RAM 
2006. It the re-extraction flag B is discriminated and this 
is found to be the initial discrimination at step S914, then 
the program returns to step S903 and re-extraction is 
executed; otherwise, the program proceeds to step 
S915, where the value of (number of small connected 
components)/(preceding extraction number) is com- 
pared with a predetermined threshold value. If the 
threshold value is larger, the program returns to step 
S903 and re-extraction is executed; otherwise, it is 
judged that-there is no background image segment and 
processing is terminated. It should be noted that in the 
case where re-extraction is performed, the background 
color range is widened. If the discrimination of the back- 
ground image segment of the inputted intermediate im- 
age segment is finished, the re-extraction flag B and the 
background color range are reset. 

Steps S916 through S920 correspond to the back- 
ground image-segment shaping step 224. At step S916, 
noise is removed from the binary image, which repre- 
sents the extracted background image segment, by ap- 
plication of well-known reduction/enlargement process- 
ing, and small image segments are removed from the 
noise-free binary image at steps S917 through S919. 
First, with regard to the noise-free binary image, a label 
image is created in the working memory and connected 
components are extracted by well-known labeling 
processingat step S91 7. The label image is one in which 
a unique number is assigned to each connected com- 
ponent and the numbers are made the pixel values of 
the connected components. Next, the number of pixels 
(the area) of each connected component is obtained 
and stored in the working memory at step S918. Next, 
at step S91 9, connected components for which the num- 
bers of pixels are less than the threshold value (namely 
connected components of small area) are eliminated. In 
other words, the values of pixels corresponding to these 
connected components of the binary image are made 
"0°s. 

Next, it is determined at step S920 whether there 
are any remaining connected components (background 
image segments). If the answer is "YES", then it is de- 
termined that there is a background image segment; 
otherwise, it is determined that there are no background 
image segments. In a case where there is a background 
image segment, a binary image indicative of a back- 
ground image segment remains in the working memory. 

Thus, in accordance with this embodiment as de- 
scribed above, a background image segmentation step 
is proposed in which it is determined whether the ex- 
tracted background image segment is appropriate or 
not, thus making it possible to avoid needless splitting 
of an image segment in a color document image having 
a portion in which there is little change in brightness and 
color, and in a color document image having a large 
amount of noise. 

In accordance with each of the embodiments de- 



scribed above, a color document image in which image 
segments having different characteristic are mixed can 
be subjected to excellent processing (image zooming, 
reduction of quantity of data and structuring of data, 
s etc.). 

<Fifth Embodiment 

An image processing apparatus according to a fifth 
io embodiment of the present invention will now be de- 
scribed. Components in the fifth embodiment similar to 
those of the first through fourth embodiments are des- 
ignated by like characters and need not be described in 
detail again. 

15 Each of the foregoing embodiments makes it pos- 
sible segment a color image into image segments in ex- 
cellent fashion. On the other hand, since document im- 
ages generally are large in size, segmentation into im- 
age segments requires a great detail of processing time. 

20 in addition, fields which utilize image segmentation 
methods employ DTP and filing systems are premised 
upon processing for carrying out dialog with the user and 
there is a need for high-speed processing that can be 
executed without hindering the processing for effecting 

25 the dialog with the user. In a case where the techniques 
of the foregoing embodiments are provided in the form 
of software and implemented by a computer, there are 
situations in which too much processing time is needed 
when the input image is large. 

30 In order to overcome this difficulty, this embodiment 
does not directly apply the techniques of the foregoing 
embodiments to the input image but applies them in- 
stead to an image obtained by reducing the input image. 
As a result, image segments are extracted at a high 

35 speed equivalent to that obtained when direct extraction 
is performed using the input image, with the extraction 
of this embodiment being based upon data indicative of 
image segments extracted from a reduced image. 
Since a reduced image is such that the size of the 

40 image to be processed is small, image-segment extrac- 
tion can be performed at high speed. Accordingly, it will 
suffice to extract an image segment by re-utilizing pa- 
rameters, which have been calculated from the reduced 
image, in the range of the reduced image segment even 

45 \t the input image is one having a large image size. As 
a result high-speed processing becomes possible. This 
means that an overall higher processing speed can be 
obtained in comparison with that achieved by directly 
applying the techniques of the foregoing embodiments 

50 to the input image. 

As an example of an image segmentation technique 
using a reduced image or a hierarchical image (a col- 
lection of a plurality of reduced images having different 
reduction ratios), see "Dialog-type Analytical Support 

55 System in Understanding of Document images, and 
Document Image Data-Base Editing Management Sys- 
tem", Kato, Hanaki, Sakai, Technical Report of the Insti- 
tute of Electronics, Information and Communication En- 
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gineers of Japan, PRL84-101, pp. 65 - 72 (1984). This 
method is for application to binary images. 

As described earlier, the color of background can 
be assumed in advance (to be white, for example) in the 
case of a binary image, but the color of background can- 
not be assumed in the case of a color image to which 
this embodiment is applied. As a consequence, there 
are situations in which image segments necessary 
when creating a reduced image are eliminated, as a re- 
sult of which image segments happen to be omitted. (In 
the case of a binary image, this can be avoided by giving 
priority to elimination of white pixels constituting the 
background.) For these and other reasons, the conven- 
tional btnarization method can be directly applied to 
color document image. 

Further, in the case of a binary image, there is little 
overlapping of image segments (or of circumscribed 
rectangles enclosing image segments) and image-seg- 
ment shapes in the input image need only be extracted 
by focusing solely upon the positions of the image seg- 
ments obtained from the reduced image (i.e., the posi- 
tions of the image segments need only be cut from the 
input image). By contrast, in the case of a color image, 
there are instances in which image segments (or cir- 
cumscribed rectangles enclosing image segments) 
overlap, and it is necessary that image-segment shapes 
in the input image be extracted upon taking into account 
not only the positions of the image segments obtained 
from the reduced image but also the shapes of the im- 
age segments obtained from the reduced image. 

[Processing procedure] 

Fig. 43 is a diagram showing a processing proce- 
dure in an image processing apparatus according to a 
fifth embodiment. A digital color image to undergo 
processing is inputted at a color image input step 1 , a 
reduced image is created from the input color image at 
a reduced-image image-segment extraction step 13, 
and an image segment is extracted from the input image 
based upon image-segment data extracted from the re- 
duced image. 

[Construction of apparatus] 

Fig. 44 is a diagram showing the construction of a 
color image processing apparatus which executes the 
processing illustrated in Fig. 43. The hardware for im- 
plementing the arrangement shown in Fig. 44 has a con- 
struction the same as that of Fig. 3 described in the first 
embodiment and need not be described again. 

In Fig. 44, the color image input unit 1001 inputs 
digital color image data to be processed and stores this 
data in the input image memory 1006. 

A reduced-image creation unit 1102 creates re- 
duced-image data from the color image data stored in 
the input image memory 1006 by the color image input 
unit 1001 and stores the created data in the a reduced- 



image memory 1106. 

A reduced-image image-segment extraction unit 
1103 extracts an image segment from the reduced im- 
age created by the reduced-image creation unit 1102 

5 and stored in the reduced-image memory 1106 and 
stores this image-segment data in the image-segment 
data memory 1007. 

An input-image image-segment extraction unit 1 1 04 
extracts an image segment in the input image from the 

10 image-segment data of the reduced image extracted by 
the reduced-image image-segment extraction unit 1103 
and stored in the image-segment data memory 1007 
and stores this image-segment data in the image-seg- 
ment data memory 1007. 

is The reduced-image creation unit 1102, the re- 
duced-image image-segment extraction unit 1103 and 
the input-image image-segment extraction unit 1104 
can be realized by the CPU 2001 , which operates in ac- 
cordance with the program stored in the ROM 2006 or 

20 storage device 2002, and the RAM 2006 or memory de- 
vice 2002 used as a working memory. It goes without 
saying that these can also be realized by their own 
CPUs, RAMs and memories or by their own hardware. 
Further, the input image memory 1006, the re- 

25 duced-image memory 1 1 06 and the image-segment da- 
ta memory 1007 can be realized by the RAM 2006 or 
memory device 2002 in Fig. 4. It goes without saying 
that these memories can be realized by their own stor- 
age devices. 

30 

[Details of processing] 

The main steps shown in Fig. 43 will now be de- 
scribed in detail while using the arrangement of Fig. 44. 

35 

Color image input step 

The color image input step 1 inputs digital color im- 
age data to be processed and stores the inputted image 
40 data in the input image memory 1006. 

Reduced-image creation step 

A reduced-image creation step 1 2 creates reduced- 
45 image data of the color image data stored in the input 
image memory 1006 by the color image input step 1 and 
stores the created data in the a reduced-image memory 
1106. Here the reduced image is obtained by reducing 
the size of the input image in the horizontal and vertical 
so directions by 1/N (where N is an arbitrary natural number 
and 1/N shall be referred to as the "reduction ratio"). The 
reduction ratio should be within a range which will not 
influence subsequent processing even through reduc- 
tion is carried out and is designated suitably depending 
S5 upon the size and content of the input image. 

The reduced-image creation step 1 2 can be imple- 
mented by a well-known method of Greating an image 
in which pixels are removed by being thinned out. An 
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example of creation of an image reduced by 1/2 in the 
vertical and horizontal directions by a well-known meth- 
od of creating an image in which pixels are removed by 
being thinned out is as follows: The Input image is divid- 
ed into blocks 31 01 of two pixels vertically and two pixels 
horizontally (four a total of four pixels) shown in Fig. 45. 
and one pixel in each block (say a pixel 31 02 in the upper 
left-hand corner) is made one corresponding pixel 3103 
of the reduced image, whereby an image reduced by 
1/2 vertically and horizontally can be created. Though 
an example has been described in which an image re- 
duced by 1/2 vertically and horizontally is created, an 
image reduced to 1/N vertically and horizontally can be 
obtained if the blocks are made three pixels in each of 
the horizontal and vertical directions (for a total of nine 
pixels), four pixels in each of the horizontal and vertical 
directions (for a total of 16 pixels), five pixels in each of 
the horizontal and vertical directions (for a total of 25 
pixels), etc. 

Further, an average value of each of the R, G, B 
values of all pixels in the block may be adopted as the 
pixel value of the reduced image. 

Reduced-image image-segment extraction step 

A reduced-image image-segment extraction step 
1 3 extracts an image segment from the reduced image 
created by the reduced-image creation unit 12 and 
stored in the reduced-image memory 1106 and stores 
this extracted image-segment data in the image-seg- 
ment data memory 1007. The reduced-image image- 
segment extraction step 1 3 can be realized by the im- 
age-segment extraction step 2 described in the first em- 
bodiment. 

Input-image image-segment extraction step 

An input-image image-segment extraction step 1 4 
creates image-segment data in an input image from im- 
age-segment data of a reduced image extracted by the 
reduced-image image-segment extraction step 1 3 and 
stored in the image-segment data memory 1007. 

Part ol the input-image image-segment creation 
step 14 is shown in Fig. 46. The manner of processing 
of the input-image image-segment creation step 14 is 
illustrated In Fig. 47. In the Input-image image-segment 
creation step 14, the process illustrated in Fig. 46 is per- 
formed with regard to all image segments that have 
been stored in the image-segment data. 

In Fig. 46, an image-segment position conversion 
step 141 calculates an image-segment position (indicat- 
ed at 4002 in Fig. 47) in the input image from an image- 
segment position (indicated at 4001 in Fig. 47) in a re- 
duced image of image-segment data. The calculated 
position is stored in the image-segment- data. More spe- 
cifically, the X, Y coordinates and values of width and 
height of the image-segment position are multiplied by 
the reciprocal N of the reduction ratio. 



An input-image image-segment shape extraction 
step 142 extracts an image-segment shape of the input 
image in a rectangular image segment of the image- 
segment position in the input image calculated at the 
5 image-segment position converting step 141 . More spe- 
cifically, a rectangular image segment of the image-seg- 
ment position calculated at the image-segment position 
conversion step 1 41 is binarized into a background color 
and image segments other than the background color, 
io a binary image (indicated at 4003 in Fig. 47) indicating 
the image-segment shape in the input image is extract- 
ed and this is stored in the image-segment shape of the 
image-segment data. The background color uses the 
value (stored as the image-segment color in the image- 
rs segment color of the background image segment) cal- 
culated at the background image segmentation step 
(see Fig. 7) in the reduced-image image-segment ex- 
traction step 1 3. As a result, the background color need 
not be re-calculated and simple binarization processed 
20 need only be applied to a limited image segment of the 
input image, thus making it possible to executed 
processing at high speed. 



The image-segment data extracted at the input-im- 
age image-segment extraction step 1 4 is capable of be- 
ing utilized in zooming of color images, reduction of the 
quantity of data and structuring of documents by the 

30 methods described in the foregoing embodiments. 

Fig. 48 is a diagram showing an example in which 
image-segment data extracted by the input-image im- 
age-segment extraction step 14 is applied to the color 
image zoom method (Fig. 2) described in connection 

3S with the first embodiment. 

Specifically, the image-segment discrimination step 
3 discriminates image-segment components (continu- 
ous color tone components, etc.) of each image seg- 
ment extracted at the input-image image-segment ex- 

40 traction step 1 4. The adaptive zooming step 4 subjects 
each image segment extracted by the input-image im- 
age-segment extraction step 14 to zoom processing 
conforming to the image-segment component discrimi- 
nated at the image-segment discrimination step 3, 

45 thereby creating a zoomed image of the input image. 
The color image output step 5 displays, makes a hard 
copy of or delivers to a communication line the zoomed 
image created by the adaptive zooming step 4. 

Fig. 49 is a diagram showing an example in which 

so image-segment data extracted by the input-image im- 
age-segment extraction step 1 4 is applied to the color 
image compression method (Fig. 25) described in con- 
nection with the second embodiment. 

Specifically, the image-segment discrimination step 

55 3 discriminates the type of each image segment extract- 
ed at the input-image image-segment extraction step 14 
and stores the result of discrimination in the image-seg- 
ment data. The adaptive compression step 6 subjects 
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each image segment extracted by the input-image im- 
age-segment extraction step 14 to compression 
processing conforming to the type of image-segment 
component discriminated at the image-segment dis- 
crimination step 3, thereby compressing the input image 
data. The compressed-data output step 7 outputs the 
compressed data created at the adaptive compression 
step 6 by transmitting the compressed data to a com- 
munication line or writing the compressed data in an im- 
age storage device. 

According to the compressed-data input step 8, the 
data compressed at the adaptive compression step 6 is 
inputted by being received from a communication line 
or read in from an image storage device. The data com- 
pressed at the adaptive compression step 6 is repro- 
duced as color image data at the color image reproduc- 
tion step 11. The image reproduced at the color image 
reproduction step 11 is displayed or produced as a hard 
copy at the color image output step 5. 

Here the steps from the color image input step 1 to 
the compressed-data output step 7 are executed when 
the color image is compressed. The steps from the com- 
pressed-data input step 8 to the color image output step 
5 are executed when image data is reproduced from the 
compressed data. 

Fig. 50 is a diagram showing an example in which 
image-segment data extracted by the input-image im- 
age-segment extraction step 14 is applied to the struc- 
turing (Fig. 35) of a document image described in con- 
nection with the third embodiment. 

Specifically, the image-segment discrimination step 
3 discriminates the type of each image segment extract- 
ed at the input-image image-segment extraction step 14 
and stores the result of discrimination in the image-seg- 
ment data. According to the data conversion step 10, 
image-segment data created by the process up to the 
image-segment discrimination step is converted, for 
each image segment, to image data having a data for- 
mat suitable for the editing of a document image in a 
DTP apparatus or the like. In the document image edit- 
ing step 1 1 , editing and output of a document image are 
performed by a well-known document image editing 
method for editing the document image in accordance 
with a designation from the user. 

<Sixth Embodiment 

An image processing apparatus according to a sixth 
embodiment of the present invention will now be de- 
scribed. Components in the sixth embodiment similar to 
those of the first through fifth embodiments are desig- 
nated by like characters and need not be described in 
detail again. The input-image image-segment extraction 
step 1 4 in the fifth embodiment binarizes a portion cor- 
responding to the image-segment position of each im- 
age segment extracted from a reduced image of the in- 
put image and calculates the image-segment shape in 
the input image. This method makes it possible to obtain 



excellent results by comparatively simple calculations. 
However, in a case where circumscribed rectangles en- 
closing extracted image segments overlap, as shown in 
Fig. 52, a problem which arises is that the shape of an- 
5 other image segment also happens to be extracted at 
the overlapping portion. 

The input-image image-segment extraction step 14 
of this embodiment solves the above-mentioned prob- 
lem and makes it possible to extract image-segment 

10 shapes correctly even in a case where the input image 
has overlapping circumscribed rectangles enclosing im- 
age segments. 

Fig. 51 is a diagram showing part of the input-image 
image-segment extraction step 14 of this embodiment. 

'5 The manner of processing of the input-image image- 
segment creation step of this embodiment is illustrated 
in Fig. 52. In the input-image image-segment extraction 
step 1 4, the process illustrated in these Figures is per- 
formed with regard to all image segments that have 

20 been stored in the image-segment data. 

In Fig. 51, the image-segment position conversion 
step 1 41 calculates an image-segment position (indicat- 
ed at 5102 in Fig. 52) in the input image from an image- 
segment position (indicated at 5101 in Fig. 52) in a re- 

25 duced image of image-segment data, as in the fifth em- 
bodiment. The calculated position is stored in the image- 
segment data. 

A reduced-image image-segment shape expansion 
step 143 applies well-known expansion processing to a 

30 binary image (indicated at 5103 in Fig. 52) indicative of 
an image-segment shape in the reduced image extract- 
ed by a reduced-image image-segment extraction step 
13. (An expanded image indicated at 5104 in Fig. 52 
shall be referred to as an "expanded image 8 .) Expansion 

35 processing is performed a number of times which is the 
reciprocal of the reduction ratio (namely N times). 

A selective input-image image-segment shape ex- 
traction step 144 extracts an image segment in the input 
image in the same manner as the input-image image- 

*o segment shape extraction step 1 42 of the fifth embodi- 
ment. According to step 144, however reference is 
made to the expanded image, which has been created 
at the reduced-image image-segment shape expansion 
step 1 43, in such a manner that extraction will not extend 

« to other image segments owing to overlapping of cir- 
cumscribed rectangles. More specifically, with regard to 
each pixel of a rectangular image segment at Image- 
segment position 5102 in the input image calculated at 
the image-segment position conversion step 1 41 , only 

50 pixels (the portions at 5105 in Fig. 52) having a value 
other than that of the background color and for which 
the corresponding pixels of the expanded image 5104 
created at the reduced-image image-segment shape 
expansion step 1 43 are pixels in the relevant image seg- 

55 ment (where the pixel values are "1 *s) are adopted as 
pixels (of value "1 °) constructing the image segment (in- 
dicated at 5106 in Fig. 52). The binary image thus cre- 
ated is stored as the image-segment shape of the im- 
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age -segment data. 

Of course, as in the fifth embodiment, the back- 
ground color uses a value calculated by the background 
image segmentation step (see Fig. 7) of the reduced- 
image image-segment extraction step 13. (The value 
has been stored as an image-segment color in the im- 
age-segment data of the background image segment.) 

It should be noted that this embodiment is capable 
of being implemented without the reduced-image im- 
age-segment shape expansion step 1 43. Though the re- 
sults of extraction will be inferior if there is no reduced- 
image image-segment shape expansion step 143, an 
advantage gained is that the process is simplified. 

In accordance with the input-image image-segment 
extraction step 14 of this embodiment, the processing 
is more complicated in comparison with the input-image 
image-segment extraction step 1 4 of the fifth embodi- 
ment. However, excellent results of image-segment ex- 
traction can be obtained even if circumscribed rectan- 
gles enclosing image segments overlap. 

<Seventh Embodiment 

An image processing apparatus according to a sev- 
enth embodiment of the present invention will now be 
described. Components in the seventh embodiment 
similar to those of the first through sixth embodiments 
are designated by like characters and need not be de- 
scribed in detail again. 

The input-image image-segment extraction step 14 
of the sixth embodiment extracts the image segments 
of an input image by combining image-segment shapes 
extracted from a reduced image and the results of bina- 
rizing the input image. This method provides excellent 
results of extraction even if circumscribed rectangles 
enclosing image segments overlap. However, when a 
color original image is reduced, there are occasions 
where the shape of an image segment in the reduced 
image is omitted owing to elimination of a pari of the 
image segment. As a consequence, the shape of an im- 
age segment in the input image finally extracted is omit- 
ted as well. 

The input-image image-segment extraction step 1 4 
of this embodiment solves the problem wherein part of 
the shape of an image segment is eliminated. 

Fig. 53 is a diagram showing part of the input-image 
image-segment extract ton step 1 4 according to this em- 
bodiment. The manner of the processing of the input- 
image image-segment extraction step 14 is illustrated 
in Fig. 54. According to the input-image image-segment 
extraction step 14, the process shown in Fig. 53 is ap- 
plied to a collection of image segments (hereinafter re- 
ferred to as "a group of image segments of the same 
level") having the same background image segment 
shown in Fig. 55, and is applied to all image segments 
that have been stored as the image-segment data. 

In Fig. 53, with regard to each image segment of 
the group of image segments of the same level, the im- 



age-segment position conversion step 141 calculates 
an image-segment position (indicated at 6002 in Fig. 54) 
in the input image from an image-segment position (in- 
dicated at 6001 in Fig. 54) in a reduced image of image- 

s segment data, as in the fifth embodiment. The calculat- 
ed position is stored in the image-segment data. 

The input-image image-segment shape extraction 
step 1 42 extracts a rectangular image segment (indicat- 
ed at 6003 in Fig. 54) corresponding to the image-seg- 

10 ment position (i.e., the image-segment position of the 
background image segment) of the group of image seg- 
ments of the same level in the input image calculated at 
the image-segment position conversion step 1 41 . In oth- 
er words, as in the input-image image-segment shape 

is extraction step 1 42 of the fifth embodiment, binarization 
is effected into a background color and image segments 
other than the background color, and an image-segment 
shape (indicated at 6004 in Fig. 54) contained in the 
group 6003 of image segments of the same level in the 

20 input image is created. Of course, as in the fifth embod- 
iment, the background color uses a value calculated by 
the background image segmentation step of the re- 
duced-image image-segment extraction step 1 3. (The 
value has been stored as an image-segment color in the 

2S image-segment data of the background image seg- 
ment.) 

An omitted image-segment extraction step 145 
compares an image segment extracted from a reduced 
image and an image segment extracted from an input 

so image, thereby extracting an area (referred to as an 
•omitted image segment") which is omitted at the time 
of reduced image creation. To this end, a number (label) 
specific to each image segment indicating whether a 
pixel is a pixel corresponding to any image segment of 

35 the group of image segments of the same level is set for 
each pixel of the binary image 6004 representing the 
image-segment shape created by the input-image im- 
age-segment shape extraction step 142. Next, in the im- 
age-segment shape of each image segment, a label (in- 

40 dicated at 6005 in Fig. 54) is written in pixels (whose 
values are "1 ") indicating the image segment in the bi- 
nary image representing this image-segment shape. Fi- 
nally, an image segment created by the input-image im- 
age-segment shape extraction step 1 42 and which has 

45 not been labeled is extracted as an omitted image seg- 
ment (indicated at 6006 in Fig. 54). 

An omitted image-segment unification step 146 uni- 
fies the omitted image segment extracted at the omitted 
image-segment extraction step 145 with image seg- 

50 ments that have already been extracted. More specifi- 
cally, a pixel not labeled at the omitted image-segment 
extraction step 1 45 is provided with the label of the near- 
est labeled pixel and is unified with image segments al- 
ready extracted (indicated at 6007 in Fig. 54). Finally, 

55 circumscribed rectangles (indicated at 6008 in Fig. 54) 
of image segments provided with the same label are ob- 
tained, the image-segment positions are changed with 
respect to the image-segment data of each image seg- 
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ment and the binary image of the pixels having the same 
label is stored as the image-segment shape. 

Thus, in accordance with the input-image image- 
segment extraction step 1 4 of this embodiment, excel- 
lent results of extraction can be obtained even if part of 
an image segment is eliminated at the time the reduced 
image is formed. 

In accordance with the fifth through seventh embod- 
iments as described above, image segmentation of a 
color document image can be performed at high speed, 
and in a manner equivalent to that in a case where im- 
age segmentation is performed directly with respect to 
an input image, regardless of the shape of the input im- 
age. A color document image can be zoomed with ex- 
cellent image quality and compression and decompres- 
sion (reproduction) can be carried out with excellent im- 
age quality and at an excellent compression ratio. In ad- 
dition, a color document image can be structured and 
the structured data can be edited. 

<Other Embodiment* 

The present invention can be applied to a system 
constituted by a plurality of devices (e.g., host computer, 
interface, reader, printer) or to an apparatus comprising 
a single device (e.g., copy machine, facsimile). 

Further, the object of the present invention can be 
also achieved by providing a storage medium storing 
program codes for performing the aforesaid processes 
to a system or an apparatus, reading the program codes 
with a computer (e.g., CPU, MPU) of the system or ap- 
paratus from the storage medium, then executing the 
program. 

In this case, the program codes read from the stor- 
age medium realize the functions according to the em- 
bodiments, and the storage medium storing the program 
codes constitutes the invention. 

Further, the storage medium, such as a floppy disk, 
a hard disk, an optical disk, a magneto-optical disk, CD- 
ROM, CD-R, a magnetic tape, a non-volatile type CD- 
ROM, CD-R, a magnetic tape, a non-volatile type mem- 
ory card, and ROM can be used for providing the pro- 
gram codes. 

Furthermore, besides aforesaid functions accord- 
ing to the above embodiments are realized by executing 
the program codes which are read by a computer, the 
present invention includes a case where an OS (Oper- 
ating System) or the like working on the computer per- 
forms a part or entire processes in accordance with des- 
ignations of the program codes and realizes functions 
according to the above embodiments. 

Furthermore, the present invention also includes a 
case where, after the program codes read from the stor- 
age medium are written in a function expansion card 
which is inserted into the computer or in a memory pro- 
vided in a function expansion unit which is connected to 
the computer, CPU or the like contained in the function 
expansion card or unit performs a part or entire process 



in accordance with designations of the program codes 
and realizes functions of the above embodiments. 

As many apparently widely different embodiments 
of the present invention can be made without departing 
s from the spirit and scope thereof, it is to be understood 
that the invention is not limited to the specific embodi- 
ments thereof except as defined in the appended 
claims. 



Claims 

I 

1. An image processing method, characterized by 
comprising: 

75 

an input step of inputting a color image; 
an extraction step of extracting an image area, 
which have a characteristic different from a 
background image area of the input color im- 
age, from said color image; and 
a discrimination step of discriminating the char- 
acteristic of the image area extracted at said 
extraction step. 

2. The method according to claim 1 , further compris- 
ing a zoom step of zooming the entered color image 
by subjecting each of the image areas extracted at 

, said extraction step to zoom processing conforming 
to results of discrimination obtained at said discrim- 
ination step. 

3. The method according to claim 1, further compris- 
ing a compression step of compressing the entered 
color image by subjecting each of the image areas 
extracted at said extraction step to compression 
processing conforming to results of discrimination 
obtained at said discrimination step. 

4. The method according to claim 3, wherein said com- 
pression step collects the compressed data of the 
image areas, obtained by compressing the data of 
each of said image areas, and adopts this data as 
compressed data of the entered color image. 

5. The method according to claim 4, further compris- 
ing a reproduction step of reproducing the color im- 
age by decompressing the compressed data per 
each image area. 

6. The method according to claim 1 , further compris- 
ing a conversion step of converting, based upon re- 
sults of discrimination obtained at said discrimina- 
tion step, the data of each of the image areas ex- 
tracted at said extraction step to a data format for 
image editing. 

7. The method according to claim 6, further compris- 
ing an editing step of editing data for image editing 
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obtained at said conversion step. 

8. The method according to claim 1 , wherein said ex- 
traction step includes: 

5 

a first separation step of separating a back* 
ground image area from an area to undergo 
processing in the entered color image; and 
a second separation step of separating at least 
one image area from image areas other than io 
said background image area; 
said extraction step extracting image areas by 
recursively repeating said first and second sep- 
aration steps in the image area separated at 
said second separation step. *5 

9. The method according to claim 8, wherein said first 
separation step detects a background portion 
based upon a distribution of pixel values in said area 

to undergo processing, separates an image area 20 
comprising pixels having values in the vicinity of the 
background pixel values and shapes the separated 
image area to make this image area said back- 
ground image area. 

25 

10. The method according to claim 9, wherein the back- 
ground pixel values are values of maximum fre- 
quency of occurrence in said area to undergo 
processing. 

30 

11. The method according to claim 8, wherein said first 
separation step detects a background portion 
based upon a distribution of pixel values in said area 
to undergo processing, separates an image area 
comprising pixels having values in the vicinity of the 35 
background pixel values, determines whether the 
separated area is appropriate as a background im- 
age area and shapes an area, which has been de- 
termined to be appropriate as a background image 
area, to make this image area said background im- 40 
age area. 

12. The method according to claim 11, wherein the 
background pixel values are values of maximum 
frequency of occurrence in said area to undergo 
processing. 

13. The method according to claim 11, wherein when 
said first separation step has determined that the 
separated image area is inappropriate as a back- so 
ground image area, said first separation step 
changes a value in the vicinity of the background 
pixel value and separates image areas again. 

14. The method according to claim 8, wherein said first ss 
separation step determines whether the separated 
image area is appropriate as a background image 
area from results of comparing the area of said area 



to undergo processing and the area of the separat- 
ed area, the area of a portion that is inside said area 
to undergo processing and outside said separated 
image area, and a number of connected compo- 
nents less than a prescribed area present outside 
the separated image area. 

15. The method according to claim 8, wherein said sec- 
ond separation step separates image areas, which 
are image areas outside said background image 
and closely adjacent to one another, as a single im- 
age area. 

16. The method according to claim 8, wherein said dis- 
crimination step retains a binary image, which indi- 
cates type, position and shape of a discriminated 
image area as characteristics of the image area, as 
well as information indicating an inclusion relation- 
ship between image areas. 

17. The method according to claim 8, wherein said dis- 
crimination step includes: 

a first determination step of determining, based 
upon peak values and variance of a saturation 
histogram of said image area, whether said im- 
age area is an image area comprising color 
components or monochromatic components; 
a second determination step which, in a case 
where a color-component image area has been 
determined at said first determination step, is 
for determining, based upon peak values and 
variance of a color histogram of said image ar- 
ea, whether the color component is a continu- 
ous color tone or a limited color; 
a third determination step which, in a case 
where a monochromatic-component image ar- 
ea has been determined at said first determina- 
tion step, is for determining, based upon peak 
values and variance of a lightness histogram of 
said image area, whether the monochromatic 
component is a continuous monochromatic 
tone or a limited value; and 
a fourth determination step of determining the 
type of said image area based upon results of 
determinations made at said second through 
fourth steps. 

18. The method according to claim 17, wherein said 
types of the image area include a continuous color 
tone, a limited-color character or line drawing, a lim- 
ited-color pseudo-halftone, a continuous mono- 
chromatic tone, a limited-value character or line 
drawing and a limited-value pseudo-halftone. 

19. The method according to claim 1 8, wherein said dis- 
crimination step retains average data within the im- 
age area in a case where the type of the image area 
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has been determined to be the limited-color char- 
acter or line drawing, the limited-color pseudo-half- 
tone, the limited-value character or line drawing and 
the limited-value pseudo-halftone. 

5 

20. An image processing apparatus, characterized by 
comprising: 

input means for inputting a color image; 
extraction means for extracting an image area, 10 
which have a characteristic different from a 
background image area of the input color im- 
age, from said color image; and 
discrimination means for discriminating the 
characteristic of the image area extracted by '5 
said extraction means. 

21. The apparatus according to claim 20, further com- 
prising zoom means for zooming the entered color 
image by subjecting each of the image areas ex- 20 
traded by said extraction means to zoom process- 
ing conforming to results of discrimination obtained 

by said discrimination means. 

22. The apparatus according to claim 20, further com- 2s 
prising compression means for compressing the en- 
tered color image by subjecting each of the image 
areas extracted by said extraction means to com- 
pression processing conforming to results of dis- 
crimination obtained by said discrimination means. 30 

23. The apparatus according to claim 22, wherein said 
compression means collects the compressed data 
of the image areas, obtained by compressing the 
data of each of said image areas, and adopts this 35 
data as compressed data of the entered color im- 
age. 

24. The apparatus according to claim 22, further com- 
prising reproduction means for reproducing the *o 
color image by decompressing the compressed da- 
ta per each image area. 

25. The apparatus according to claim 20, further com- 
prising conversion means for converting, based up- 
on results of discrimination obtained by said dis- 
crimination means, the data of each of the image 
areas extracted by said extraction means to a data 
format for image editing. 

so 

26. The apparatus accbrding to claim 25, further com- 
prising editing means for editing data for image ed- 
iting obtained by said conversion means. 

27. An image processing method for extracting an im- ss 
age area from an image in which image areas hav- 
ing different components are mixed, characterized 

by comprising: 



a reduction step of generating a reduced image 
from an input image; 

a first extraction step of extracting an image ar- 
ea from the reduced image; and 
a second extraction step of extracting an image 
area from the input image using extraction re- 
sult of said first extraction step. 

28. The method according to claim 27, wherein said first 
extraction step includes: 

a first separation step of determining whether 
an image area to undergo processing in the in- 
put color image has an image area which will 
be background, and separating this back- 
ground image area; and 
a second separation step of separating at least 
one image area from image areas other than 
said background image area; 
said first extraction step extracting image areas 
by recursively repeating said first and second 
separation steps in the image area separated 
at said second separation step. 

29. The method according to claim 27, wherein said 
second extraction step includes: 

an image-area position conversion step of cal- 
culating position information of an image area 
in the input color image from position informa- 
tion of the image area extracted at said first ex- 
traction step and a reduction ratio of the re- 
duced image; and 

an image-area shape extraction step of extract- 
ing an image-area shape of the input color im- 
age based upon a rectangular image area at 
the image-area position calculated at said im- 
age-area position conversion step. 

30. The method according to claim 27, wherein said 
second extraction step includes: 

an image-area position conversion step of cal- 
culating position information of an image area 
in the input color image from position informa- 
tion of the image area extracted at said first ex- 
traction step and a reduction ratio of the re- 
duced image; and 

an image-area shape extraction step of extract- 
ing an image-area shape in the input color im- 
age from pixels of the input color image that cor- 
respond to the image area extracted at said first 
extraction step. 

31. The method according to claim 27, wherein said 
second extraction step includes: 

an image-area position conversion step of cal- 
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culating position information of an image area 
in the input color image from position informa- 
tion of the image area extracted at said first ex- 
traction step and a reduction ratio of the re- 
duced image; 

an image-area shape expansion step of apply- 
ing expansion processing to the image-area 
shape extracted at said first extraction step; 
and 

an image-area shape extraction step of extract- 
ing an image-area shape in the input color im- 
age from pixels of the input color image that cor- 
respond to the image area expanded at said im- 
age-area shape expansion step. 

32. The method according to claim 27, wherein said 
second extraction step includes: 

an image-area position conversion step of cal- 
culating position information of an image area 
in the input color image from position informa- 
tion of the image area extracted at said first ex- 
traction step and a reduction ratio of the re- 
duced image; 

an image-area shape extraction step of extract- 
ing an image-area shape of the input color im- 
age in a rectangular image area at the image- 
area position calculated at said image-area po- 
sition conversion step; 

an omitted image-area extraction step of ex- 
tracting an image area, which has been omitted 
by processing at said creation step, by compar- 
ing the image area extracted from the reduced 
image and the image area extracted from the 
input color image; and 

a unification step of unifying the image-area 
shape extracted at said image-area shape ex- 
traction step and the omitted image area ex- 
tracted at said omitted image-area extraction 
step. 

33. The method according to claim 32, wherein said 
omitted image-area extraction step assigns labels, 
which indicate respective ones of image areas ex- 
tracted at said first extraction step, to respective 
ones of pixels corresponding to the image-area 
shape extracted at said image-area shape extrac- 
tion step, and extracts pixels, to which said labels 
could not be assigned, as said omitted image area; 
and 

said unification step unifies said omitted im- 
age area by assigning each pixel, which corre- 
sponds to said omitted image area, the label of the 
nearest pixel to which a label has been assigned. 

34. The method according to claim 29, wherein said 
second extraction step and said image-area shape 
extraction step binarize the input color image to 



56 

background and portions other than background 
using the value of the background color calculated 
at said first extraction step, and extracts an image 
area in the input color image from the binary image. 

5 

35. The method according to claim 27, wherein said 
creation step selects one pixel from each block 
comprising N x N (where N is a natural number) 
pixels obtained by partitioning the input color image, 

10 and representing this block by the one pixel, thereby 
creating a reduced image having a reduction ratio 
of 1/N. 

36. The method according to claim 27, wherein said 
is creation step obtains an average value of pixels of 

each block comprising N x N (where N is a natural 
number) pixels obtained by partitioning the input 
color image, and representing this block by the av- 
erage value, thereby creating a reduced image hav- 
20 ing a reduction ratio of 1/N. 

37. The method according to claim 27, further compris- 
ing: 

2S a discrimination step of discriminating a com- 

ponent of the image area extracted at said sec- 
ond extraction step; and 
a zoom step of zooming the input color image 
by zooming the data of each area extracted at 

30 said second extraction step using a zoom meth- 

od conforming to results of discrimination ob- 
tained at said discrimination step. 

38. The method according to claim 27, further compris- 
35 ing: 

a discrimination step of discriminating a com- 
ponent of the image area extracted at said sec- 
ond extraction step; and 

40 a compression step of compressing the input 

color image by compressing the data of each 
area extracted at said second extraction step 
using a compression method conforming to re- 
suits of discrimination obtained at said discrim- 

45 ination step. 

39. The method according to claim 38, further compris- 
ing a reproduction step of reproducing the color im- 
age by decompressing the compressed data, which 

so has been obtained at said compression step, per 
each image area. 

40. The method according to claim 27, further compris- 
ing: 

55 

a discrimination step of discriminating a com- 
ponent of the image area extracted at said sec- 
ond extraction step; and 
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41 



a conversion step of converting, based upon re- 
sults of discrimination obtained at said discrim- 
ination step, the data of the image area extract- 
ed at said second extraction step to a data for- 
mat for image editing. 

The method according to claim 40, further compris- 
ing an editing step of editing data for image editing 
obtained at said conversion step. 



42. An image processing apparatus for extracting an 
image area from an image in which image areas 
having different components are mixed, character- 
ized by comprising: 

reduction means for generating a reduced im- 
age from an input image; 
first extraction means for extracting an image 
area from the reduced image; and 
second extraction means for extracting an im- 
age area from the input image using extraction 
result of said first extraction means. 



w 



is 



ponent of the image area extracted by said sec- 
ond extraction means; and 
conversion means for converting, based upon 
results of discrimination obtained by said dis- 
crimination means, the data of the image area 
extracted by said second extraction means to 
a data format for image editing. 

47. The apparatus according to claim 46, further com- 
prising editing means for editing data for image 
editing obtained by said conversion means. 



43. The apparatus according to claim 42, further com- 
prising: 25 

discrimination means for discriminating a com- 
ponent of the image area extracted by said sec- 
ond extraction means; and 

zoom means for zooming the input color image 30 
by zooming the data of each area extracted by 
said second extraction means using a zoom 
method conforming to results of discrimination 
obtained by said discrimination means. 

35 

44. The apparatus according to claim 42, further com- 
prising: 

discrimination means for discriminating a com- 
ponent of the image area extracted by said sec- 40 
ond extraction means; and 
compression means for compressing the input 
color image by compressing the data of each 
area extracted by said second extraction 
means using a compression method conform- 45 
ing to results of discrimination obtained by said 
discrimination means. 



45. The method according to claim 44, further compris- 
ing reproduction means for reproducing the color so 
image by decompressing the compressed data, 
which has been obtained by said compression 
means, per each image area. 

46. The apparatus according to claim 42, further com- 55 
prising: 

discrimination means for discriminating a com- 
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extraction step, and the data of the extracted image seg- 
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